You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by "[mRg]" <em...@gmail.com> on 2010/09/09 15:25:23 UTC
Couch not releasing deleted files
[CouchDB Version : 0.11]
[OS: RHEL 5.5]
Hi all,
We are seeing some odd behaviour with CouchDb not releasing files after it
has compacted them meaning we are seeing disk space continue to increase
until the service is restarted even though we have regular compaction /
cleanup.
Typical scenario : File system alert for disk space ..
df says 5.5.GB used ..
df -h | grep -P 'Filesystem|db'
Filesystem Size Used Avail Use% Mounted on
8.0G 5.5G 2.2G 72% /var/lib/db
.. but the files on the disk (and in Futon) say only 793MB is being used ..
du -sh /var/lib/db/*
793M /var/lib/db/couchdb
.. running lsof shows the files marked as deleted but still there (I'm
assuming beam still has some kind of handle still open to them) ..
lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
COMMAND PID USER FD TYPE DEVICE SIZE
NODE NAME
beam.smp 21352 couchdb 12u REG 253,10 4183
1210374
/var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
beam.smp 21352 couchdb 16u REG 253,10 612039080
1204227 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 20u REG 253,10 180699242
1204237 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 21u REG 253,10 3133540
2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
beam.smp 21352 couchdb 25u REG 253,10 71422052
2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 26u REG 253,10 71372900
2056193 /var/lib/db/couchdb/active_context_schedule.couch
beam.smp 21352 couchdb 28u REG 253,10 621252717
1204238 /var/lib/db/couchdb/active_context.couch
beam.smp 21352 couchdb 29u REG 253,10 1740903
90113 /var/lib/db/couchdb/mapping_cache.couch
beam.smp 21352 couchdb 30u REG 253,10 3133540
1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
beam.smp 21352 couchdb 31u REG 253,10 2318439
1204240 /var/lib/db/couchdb/mappings.couch (deleted)
beam.smp 21352 couchdb 32u REG 253,10 983150
2004994
/var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
beam.smp 21352 couchdb 33u REG 253,10 3133540
2056194 /var/lib/db/couchdb/thesaurus.couch
beam.smp 21352 couchdb 34u REG 253,10 70742116
1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 44u REG 253,10 1740903
2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
beam.smp 21352 couchdb 45u REG 253,10 65396836
1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 47u REG 253,10 71229540
1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 48u REG 253,10 240254078
1204226 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 49u REG 253,10 2232423
1204241 /var/lib/db/couchdb/mappings.couch
beam.smp 21352 couchdb 51u REG 253,10 96903353
1210371
/var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
beam.smp 21352 couchdb 52u REG 253,10 1192034
1249282
/var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
beam.smp 21352 couchdb 53u REG 253,10 1622135
2004993
/var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
beam.smp 21352 couchdb 54u REG 253,10 4172
1208327
/var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
beam.smp 21352 couchdb 55u REG 253,10 13496483
1208325
/var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
beam.smp 21352 couchdb 56u REG 253,10 12959902
1208326
/var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
beam.smp 21352 couchdb 57u REG 253,10 70254692
1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 58u REG 253,10 581730
1210373
/var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
beam.smp 21352 couchdb 67u REG 253,10 1818727
1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
beam.smp 21352 couchdb 78u REG 253,10 609905146
1204234 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 80u REG 253,10 545919085
1994753 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 81u REG 253,10 71032932
1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 84u REG 253,10 548866111
1204239 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 85u REG 253,10 564612402
1204233 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 87u REG 253,10 615996179
1204228 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 102u REG 253,10 561111149
2023425 /var/lib/db/couchdb/active_context.couch (deleted)
Has anyone seen this behaviour before or knows of a way around this ?
Currently we have to restart the couchdb service in order to restore the
disk space.
Any help/advice is much appreciated.
Regards
Stephen
Re: Couch not releasing deleted files
Posted by Filipe David Manana <fd...@apache.org>.
Last week I was able to reproduce and fixed it.
https://issues.apache.org/jira/browse/COUCHDB-926 is now closed.
The same type of issue happened with continuous replications and the
the authentication cache (_users database).
Both fixed in trunk, 1.1.x and 1.0.x.
1.0.2 and 1.1.0 will contain the fixes.
On Tue, Sep 28, 2010 at 1:27 AM, Panop S. <pa...@vizrt.com> wrote:
> Hi,
>
> Thanks, how to apply this ?
> By replacing this couchdb/trunk/src/couchdb/couch_file.erl file or I
> have to build it all again ?
>
> Bests,
> Aun.
>
> On Mon, 2010-09-27 at 12:14 +0200, [mRg] wrote:
>> Anyone still following this on here, I asked this question on the dev list
>> and Filipe has committed a potential fix for this issue here:
>>
>> http://svn.apache.org/viewvc?view=revision&revision=1001196
>>
>> <http://svn.apache.org/viewvc?view=revision&revision=1001196>if people are
>> still having this issue they could try applying this, we are now doing
>> internal testing to see if this has fixed it for our use case.
>>
>> Regards
>>
>> Stephen
>>
>> On 19 September 2010 00:36, Robert Newson <ro...@gmail.com> wrote:
>>
>> > It's been discussed by dev's at CouchCamp but a mailing list reminder
>> > couldn't hurt. There's some discussion about 1.1, so perhaps that's a
>> > good thread to raise this on.
>> >
>> > B.
>> >
>> > On Sun, Sep 19, 2010 at 12:24 AM, [mRg] <em...@gmail.com> wrote:
>> > > Has this been raised with the dev list or is there a bug raised for this
>> > > issue ? Seems a few people are having this and more are probably not
>> > seeing
>> > > yet.
>> > >
>> > > On 15 September 2010 08:49, [mRg] <em...@gmail.com> wrote:
>> > >
>> > >> Yes there are 6 views (2 design docs)
>> > >>
>> > >>
>> > >> On 14 September 2010 21:51, Adam Kocoloski <ko...@apache.org> wrote:
>> > >>
>> > >>> Were there any view indexes on that DB? My suspicion is that
>> > >>> couch_view_group misses a step somewhere in its dance and keeps a
>> > reference
>> > >>> to an old #db{}
>> > >>>
>> > >>> Adam
>> > >>>
>> > >>> On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
>> > >>>
>> > >>> > It was just on a DB with a lot of writes (documents having lots of
>> > >>> > revisions, rather that lots of new docs), the database grows as
>> > expected
>> > >>> but
>> > >>> > after just doing a simple compact on the db the old files still seem
>> > to
>> > >>> be
>> > >>> > on disk as reported previously.
>> > >>> >
>> > >>> > It seems its been around a while now as (as far as we can see) 0.11 /
>> > >>> 1.0
>> > >>> > and 1.0.1 all have the same issue, when we replicated the same data
>> > over
>> > >>> to
>> > >>> > the newer versions the same identical problem happened.
>> > >>> >
>> > >>> > On 14 September 2010 14:49, Robert Newson <ro...@gmail.com>
>> > >>> wrote:
>> > >>> >
>> > >>> >> This was mentioned at CouchCamp, several developers suspect there's
>> > a
>> > >>> >> subtle bug in the code where file descriptors are not being closed
>> > >>> >> under some conditions.
>> > >>> >>
>> > >>> >> Did the OP identify a particular sequence that led here?
>> > >>> >>
>> > >>> >> B.
>> > >>> >>
>> > >>> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <
>> > >>> juanjo@comellas.org>
>> > >>> >> wrote:
>> > >>> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the
>> > code,
>> > >>> but
>> > >>> >>> what is happening is that CouchDB is keeping file descriptors
>> > >>> >> corresponding
>> > >>> >>> to deleted files open, probably after compacting a database. I'm
>> > >>> >> inferring
>> > >>> >>> this based on the fact that with lsof I saw several cases of two
>> > files
>> > >>> >> with
>> > >>> >>> the same name (one deleted and one not) opened by CouchDB.
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
>> > >>> >>>
>> > >>> >>>> Is this an issue with the OS or with Couch/Beam itself ?
>> > >>> >>>>
>> > >>> >>>> On 10 September 2010 16:57, Juan Jose Comellas <
>> > juanjo@comellas.org>
>> > >>> >>>> wrote:
>> > >>> >>>>
>> > >>> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The
>> > only
>> > >>> >> way
>> > >>> >>>> I
>> > >>> >>>>> found to solve the problem was by restarting CouchDB.
>> > >>> >>>>>
>> > >>> >>>>>
>> > >>> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com>
>> > wrote:
>> > >>> >>>>>
>> > >>> >>>>>> [CouchDB Version : 0.11]
>> > >>> >>>>>> [OS: RHEL 5.5]
>> > >>> >>>>>>
>> > >>> >>>>>> Hi all,
>> > >>> >>>>>>
>> > >>> >>>>>> We are seeing some odd behaviour with CouchDb not releasing
>> > files
>> > >>> >> after
>> > >>> >>>>> it
>> > >>> >>>>>> has compacted them meaning we are seeing disk space continue to
>> > >>> >>>> increase
>> > >>> >>>>>> until the service is restarted even though we have regular
>> > >>> >> compaction /
>> > >>> >>>>>> cleanup.
>> > >>> >>>>>>
>> > >>> >>>>>> Typical scenario : File system alert for disk space ..
>> > >>> >>>>>>
>> > >>> >>>>>> df says 5.5.GB used ..
>> > >>> >>>>>>
>> > >>> >>>>>> df -h | grep -P 'Filesystem|db'
>> > >>> >>>>>>
>> > >>> >>>>>> Filesystem Size Used Avail Use% Mounted on
>> > >>> >>>>>>
>> > >>> >>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> .. but the files on the disk (and in Futon) say only 793MB is
>> > being
>> > >>> >>>> used
>> > >>> >>>>> ..
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> du -sh /var/lib/db/*
>> > >>> >>>>>>
>> > >>> >>>>>> 793M /var/lib/db/couchdb
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> .. running lsof shows the files marked as deleted but still
>> > there
>> > >>> >> (I'm
>> > >>> >>>>>> assuming beam still has some kind of handle still open to them)
>> > ..
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>> > >>> >>>>>>
>> > >>> >>>>>> COMMAND PID USER FD TYPE DEVICE
>> > >>> >> SIZE
>> > >>> >>>>>> NODE NAME
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 12u REG 253,10
>> > >>> >> 4183
>> > >>> >>>>>> 1210374
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 16u REG 253,10
>> > >>> >> 612039080
>> > >>> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 20u REG 253,10
>> > >>> >> 180699242
>> > >>> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 21u REG 253,10
>> > >>> >> 3133540
>> > >>> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 25u REG 253,10
>> > >>> >> 71422052
>> > >>> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 26u REG 253,10
>> > >>> >> 71372900
>> > >>> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 28u REG 253,10
>> > >>> >> 621252717
>> > >>> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 29u REG 253,10
>> > >>> >> 1740903
>> > >>> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 30u REG 253,10
>> > >>> >> 3133540
>> > >>> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 31u REG 253,10
>> > >>> >> 2318439
>> > >>> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 32u REG 253,10
>> > >>> >> 983150
>> > >>> >>>>>> 2004994
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 33u REG 253,10
>> > >>> >> 3133540
>> > >>> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 34u REG 253,10
>> > >>> >> 70742116
>> > >>> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 44u REG 253,10
>> > >>> >> 1740903
>> > >>> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 45u REG 253,10
>> > >>> >> 65396836
>> > >>> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 47u REG 253,10
>> > >>> >> 71229540
>> > >>> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 48u REG 253,10
>> > >>> >> 240254078
>> > >>> >>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 49u REG 253,10
>> > >>> >> 2232423
>> > >>> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 51u REG 253,10
>> > >>> >> 96903353
>> > >>> >>>>>> 1210371
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 52u REG 253,10
>> > >>> >> 1192034
>> > >>> >>>>>> 1249282
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 53u REG 253,10
>> > >>> >> 1622135
>> > >>> >>>>>> 2004993
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 54u REG 253,10
>> > >>> >> 4172
>> > >>> >>>>>> 1208327
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 55u REG 253,10
>> > >>> >> 13496483
>> > >>> >>>>>> 1208325
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 56u REG 253,10
>> > >>> >> 12959902
>> > >>> >>>>>> 1208326
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 57u REG 253,10
>> > >>> >> 70254692
>> > >>> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 58u REG 253,10
>> > >>> >> 581730
>> > >>> >>>>>> 1210373
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 67u REG 253,10
>> > >>> >> 1818727
>> > >>> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 78u REG 253,10
>> > >>> >> 609905146
>> > >>> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 80u REG 253,10
>> > >>> >> 545919085
>> > >>> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 81u REG 253,10
>> > >>> >> 71032932
>> > >>> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 84u REG 253,10
>> > >>> >> 548866111
>> > >>> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 85u REG 253,10
>> > >>> >> 564612402
>> > >>> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 87u REG 253,10
>> > >>> >> 615996179
>> > >>> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp 21352 couchdb 102u REG 253,10
>> > >>> >> 561111149
>> > >>> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> Has anyone seen this behaviour before or knows of a way around
>> > this
>> > >>> >> ?
>> > >>> >>>>>> Currently we have to restart the couchdb service in order to
>> > >>> restore
>> > >>> >>>> the
>> > >>> >>>>>> disk space.
>> > >>> >>>>>>
>> > >>> >>>>>> Any help/advice is much appreciated.
>> > >>> >>>>>>
>> > >>> >>>>>> Regards
>> > >>> >>>>>>
>> > >>> >>>>>> Stephen
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>>
>> > >>> >>
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>
>
>
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: Couch not releasing deleted files
Posted by "Panop S." <pa...@vizrt.com>.
Hi,
Thanks, how to apply this ?
By replacing this couchdb/trunk/src/couchdb/couch_file.erl file or I
have to build it all again ?
Bests,
Aun.
On Mon, 2010-09-27 at 12:14 +0200, [mRg] wrote:
> Anyone still following this on here, I asked this question on the dev list
> and Filipe has committed a potential fix for this issue here:
>
> http://svn.apache.org/viewvc?view=revision&revision=1001196
>
> <http://svn.apache.org/viewvc?view=revision&revision=1001196>if people are
> still having this issue they could try applying this, we are now doing
> internal testing to see if this has fixed it for our use case.
>
> Regards
>
> Stephen
>
> On 19 September 2010 00:36, Robert Newson <ro...@gmail.com> wrote:
>
> > It's been discussed by dev's at CouchCamp but a mailing list reminder
> > couldn't hurt. There's some discussion about 1.1, so perhaps that's a
> > good thread to raise this on.
> >
> > B.
> >
> > On Sun, Sep 19, 2010 at 12:24 AM, [mRg] <em...@gmail.com> wrote:
> > > Has this been raised with the dev list or is there a bug raised for this
> > > issue ? Seems a few people are having this and more are probably not
> > seeing
> > > yet.
> > >
> > > On 15 September 2010 08:49, [mRg] <em...@gmail.com> wrote:
> > >
> > >> Yes there are 6 views (2 design docs)
> > >>
> > >>
> > >> On 14 September 2010 21:51, Adam Kocoloski <ko...@apache.org> wrote:
> > >>
> > >>> Were there any view indexes on that DB? My suspicion is that
> > >>> couch_view_group misses a step somewhere in its dance and keeps a
> > reference
> > >>> to an old #db{}
> > >>>
> > >>> Adam
> > >>>
> > >>> On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
> > >>>
> > >>> > It was just on a DB with a lot of writes (documents having lots of
> > >>> > revisions, rather that lots of new docs), the database grows as
> > expected
> > >>> but
> > >>> > after just doing a simple compact on the db the old files still seem
> > to
> > >>> be
> > >>> > on disk as reported previously.
> > >>> >
> > >>> > It seems its been around a while now as (as far as we can see) 0.11 /
> > >>> 1.0
> > >>> > and 1.0.1 all have the same issue, when we replicated the same data
> > over
> > >>> to
> > >>> > the newer versions the same identical problem happened.
> > >>> >
> > >>> > On 14 September 2010 14:49, Robert Newson <ro...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> >> This was mentioned at CouchCamp, several developers suspect there's
> > a
> > >>> >> subtle bug in the code where file descriptors are not being closed
> > >>> >> under some conditions.
> > >>> >>
> > >>> >> Did the OP identify a particular sequence that led here?
> > >>> >>
> > >>> >> B.
> > >>> >>
> > >>> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <
> > >>> juanjo@comellas.org>
> > >>> >> wrote:
> > >>> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the
> > code,
> > >>> but
> > >>> >>> what is happening is that CouchDB is keeping file descriptors
> > >>> >> corresponding
> > >>> >>> to deleted files open, probably after compacting a database. I'm
> > >>> >> inferring
> > >>> >>> this based on the fact that with lsof I saw several cases of two
> > files
> > >>> >> with
> > >>> >>> the same name (one deleted and one not) opened by CouchDB.
> > >>> >>>
> > >>> >>>
> > >>> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
> > >>> >>>
> > >>> >>>> Is this an issue with the OS or with Couch/Beam itself ?
> > >>> >>>>
> > >>> >>>> On 10 September 2010 16:57, Juan Jose Comellas <
> > juanjo@comellas.org>
> > >>> >>>> wrote:
> > >>> >>>>
> > >>> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The
> > only
> > >>> >> way
> > >>> >>>> I
> > >>> >>>>> found to solve the problem was by restarting CouchDB.
> > >>> >>>>>
> > >>> >>>>>
> > >>> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com>
> > wrote:
> > >>> >>>>>
> > >>> >>>>>> [CouchDB Version : 0.11]
> > >>> >>>>>> [OS: RHEL 5.5]
> > >>> >>>>>>
> > >>> >>>>>> Hi all,
> > >>> >>>>>>
> > >>> >>>>>> We are seeing some odd behaviour with CouchDb not releasing
> > files
> > >>> >> after
> > >>> >>>>> it
> > >>> >>>>>> has compacted them meaning we are seeing disk space continue to
> > >>> >>>> increase
> > >>> >>>>>> until the service is restarted even though we have regular
> > >>> >> compaction /
> > >>> >>>>>> cleanup.
> > >>> >>>>>>
> > >>> >>>>>> Typical scenario : File system alert for disk space ..
> > >>> >>>>>>
> > >>> >>>>>> df says 5.5.GB used ..
> > >>> >>>>>>
> > >>> >>>>>> df -h | grep -P 'Filesystem|db'
> > >>> >>>>>>
> > >>> >>>>>> Filesystem Size Used Avail Use% Mounted on
> > >>> >>>>>>
> > >>> >>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>> .. but the files on the disk (and in Futon) say only 793MB is
> > being
> > >>> >>>> used
> > >>> >>>>> ..
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>> du -sh /var/lib/db/*
> > >>> >>>>>>
> > >>> >>>>>> 793M /var/lib/db/couchdb
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>> .. running lsof shows the files marked as deleted but still
> > there
> > >>> >> (I'm
> > >>> >>>>>> assuming beam still has some kind of handle still open to them)
> > ..
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
> > >>> >>>>>>
> > >>> >>>>>> COMMAND PID USER FD TYPE DEVICE
> > >>> >> SIZE
> > >>> >>>>>> NODE NAME
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 12u REG 253,10
> > >>> >> 4183
> > >>> >>>>>> 1210374
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 16u REG 253,10
> > >>> >> 612039080
> > >>> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 20u REG 253,10
> > >>> >> 180699242
> > >>> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 21u REG 253,10
> > >>> >> 3133540
> > >>> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 25u REG 253,10
> > >>> >> 71422052
> > >>> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch
> > (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 26u REG 253,10
> > >>> >> 71372900
> > >>> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 28u REG 253,10
> > >>> >> 621252717
> > >>> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 29u REG 253,10
> > >>> >> 1740903
> > >>> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 30u REG 253,10
> > >>> >> 3133540
> > >>> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 31u REG 253,10
> > >>> >> 2318439
> > >>> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 32u REG 253,10
> > >>> >> 983150
> > >>> >>>>>> 2004994
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 33u REG 253,10
> > >>> >> 3133540
> > >>> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 34u REG 253,10
> > >>> >> 70742116
> > >>> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch
> > (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 44u REG 253,10
> > >>> >> 1740903
> > >>> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 45u REG 253,10
> > >>> >> 65396836
> > >>> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch
> > (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 47u REG 253,10
> > >>> >> 71229540
> > >>> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch
> > (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 48u REG 253,10
> > >>> >> 240254078
> > >>> >>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 49u REG 253,10
> > >>> >> 2232423
> > >>> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 51u REG 253,10
> > >>> >> 96903353
> > >>> >>>>>> 1210371
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 52u REG 253,10
> > >>> >> 1192034
> > >>> >>>>>> 1249282
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 53u REG 253,10
> > >>> >> 1622135
> > >>> >>>>>> 2004993
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 54u REG 253,10
> > >>> >> 4172
> > >>> >>>>>> 1208327
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 55u REG 253,10
> > >>> >> 13496483
> > >>> >>>>>> 1208325
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 56u REG 253,10
> > >>> >> 12959902
> > >>> >>>>>> 1208326
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 57u REG 253,10
> > >>> >> 70254692
> > >>> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch
> > (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 58u REG 253,10
> > >>> >> 581730
> > >>> >>>>>> 1210373
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>
> > >>>
> > /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 67u REG 253,10
> > >>> >> 1818727
> > >>> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 78u REG 253,10
> > >>> >> 609905146
> > >>> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 80u REG 253,10
> > >>> >> 545919085
> > >>> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 81u REG 253,10
> > >>> >> 71032932
> > >>> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch
> > (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 84u REG 253,10
> > >>> >> 548866111
> > >>> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 85u REG 253,10
> > >>> >> 564612402
> > >>> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 87u REG 253,10
> > >>> >> 615996179
> > >>> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>> beam.smp 21352 couchdb 102u REG 253,10
> > >>> >> 561111149
> > >>> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>> Has anyone seen this behaviour before or knows of a way around
> > this
> > >>> >> ?
> > >>> >>>>>> Currently we have to restart the couchdb service in order to
> > >>> restore
> > >>> >>>> the
> > >>> >>>>>> disk space.
> > >>> >>>>>>
> > >>> >>>>>> Any help/advice is much appreciated.
> > >>> >>>>>>
> > >>> >>>>>> Regards
> > >>> >>>>>>
> > >>> >>>>>> Stephen
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>>
> > >>> >>
> > >>>
> > >>>
> > >>
> > >
> >
Re: Couch not releasing deleted files
Posted by "[mRg]" <em...@gmail.com>.
Anyone still following this on here, I asked this question on the dev list
and Filipe has committed a potential fix for this issue here:
http://svn.apache.org/viewvc?view=revision&revision=1001196
<http://svn.apache.org/viewvc?view=revision&revision=1001196>if people are
still having this issue they could try applying this, we are now doing
internal testing to see if this has fixed it for our use case.
Regards
Stephen
On 19 September 2010 00:36, Robert Newson <ro...@gmail.com> wrote:
> It's been discussed by dev's at CouchCamp but a mailing list reminder
> couldn't hurt. There's some discussion about 1.1, so perhaps that's a
> good thread to raise this on.
>
> B.
>
> On Sun, Sep 19, 2010 at 12:24 AM, [mRg] <em...@gmail.com> wrote:
> > Has this been raised with the dev list or is there a bug raised for this
> > issue ? Seems a few people are having this and more are probably not
> seeing
> > yet.
> >
> > On 15 September 2010 08:49, [mRg] <em...@gmail.com> wrote:
> >
> >> Yes there are 6 views (2 design docs)
> >>
> >>
> >> On 14 September 2010 21:51, Adam Kocoloski <ko...@apache.org> wrote:
> >>
> >>> Were there any view indexes on that DB? My suspicion is that
> >>> couch_view_group misses a step somewhere in its dance and keeps a
> reference
> >>> to an old #db{}
> >>>
> >>> Adam
> >>>
> >>> On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
> >>>
> >>> > It was just on a DB with a lot of writes (documents having lots of
> >>> > revisions, rather that lots of new docs), the database grows as
> expected
> >>> but
> >>> > after just doing a simple compact on the db the old files still seem
> to
> >>> be
> >>> > on disk as reported previously.
> >>> >
> >>> > It seems its been around a while now as (as far as we can see) 0.11 /
> >>> 1.0
> >>> > and 1.0.1 all have the same issue, when we replicated the same data
> over
> >>> to
> >>> > the newer versions the same identical problem happened.
> >>> >
> >>> > On 14 September 2010 14:49, Robert Newson <ro...@gmail.com>
> >>> wrote:
> >>> >
> >>> >> This was mentioned at CouchCamp, several developers suspect there's
> a
> >>> >> subtle bug in the code where file descriptors are not being closed
> >>> >> under some conditions.
> >>> >>
> >>> >> Did the OP identify a particular sequence that led here?
> >>> >>
> >>> >> B.
> >>> >>
> >>> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <
> >>> juanjo@comellas.org>
> >>> >> wrote:
> >>> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the
> code,
> >>> but
> >>> >>> what is happening is that CouchDB is keeping file descriptors
> >>> >> corresponding
> >>> >>> to deleted files open, probably after compacting a database. I'm
> >>> >> inferring
> >>> >>> this based on the fact that with lsof I saw several cases of two
> files
> >>> >> with
> >>> >>> the same name (one deleted and one not) opened by CouchDB.
> >>> >>>
> >>> >>>
> >>> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
> >>> >>>
> >>> >>>> Is this an issue with the OS or with Couch/Beam itself ?
> >>> >>>>
> >>> >>>> On 10 September 2010 16:57, Juan Jose Comellas <
> juanjo@comellas.org>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The
> only
> >>> >> way
> >>> >>>> I
> >>> >>>>> found to solve the problem was by restarting CouchDB.
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com>
> wrote:
> >>> >>>>>
> >>> >>>>>> [CouchDB Version : 0.11]
> >>> >>>>>> [OS: RHEL 5.5]
> >>> >>>>>>
> >>> >>>>>> Hi all,
> >>> >>>>>>
> >>> >>>>>> We are seeing some odd behaviour with CouchDb not releasing
> files
> >>> >> after
> >>> >>>>> it
> >>> >>>>>> has compacted them meaning we are seeing disk space continue to
> >>> >>>> increase
> >>> >>>>>> until the service is restarted even though we have regular
> >>> >> compaction /
> >>> >>>>>> cleanup.
> >>> >>>>>>
> >>> >>>>>> Typical scenario : File system alert for disk space ..
> >>> >>>>>>
> >>> >>>>>> df says 5.5.GB used ..
> >>> >>>>>>
> >>> >>>>>> df -h | grep -P 'Filesystem|db'
> >>> >>>>>>
> >>> >>>>>> Filesystem Size Used Avail Use% Mounted on
> >>> >>>>>>
> >>> >>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> .. but the files on the disk (and in Futon) say only 793MB is
> being
> >>> >>>> used
> >>> >>>>> ..
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> du -sh /var/lib/db/*
> >>> >>>>>>
> >>> >>>>>> 793M /var/lib/db/couchdb
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> .. running lsof shows the files marked as deleted but still
> there
> >>> >> (I'm
> >>> >>>>>> assuming beam still has some kind of handle still open to them)
> ..
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
> >>> >>>>>>
> >>> >>>>>> COMMAND PID USER FD TYPE DEVICE
> >>> >> SIZE
> >>> >>>>>> NODE NAME
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 12u REG 253,10
> >>> >> 4183
> >>> >>>>>> 1210374
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 16u REG 253,10
> >>> >> 612039080
> >>> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 20u REG 253,10
> >>> >> 180699242
> >>> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 21u REG 253,10
> >>> >> 3133540
> >>> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 25u REG 253,10
> >>> >> 71422052
> >>> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch
> (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 26u REG 253,10
> >>> >> 71372900
> >>> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 28u REG 253,10
> >>> >> 621252717
> >>> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 29u REG 253,10
> >>> >> 1740903
> >>> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 30u REG 253,10
> >>> >> 3133540
> >>> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 31u REG 253,10
> >>> >> 2318439
> >>> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 32u REG 253,10
> >>> >> 983150
> >>> >>>>>> 2004994
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 33u REG 253,10
> >>> >> 3133540
> >>> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 34u REG 253,10
> >>> >> 70742116
> >>> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch
> (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 44u REG 253,10
> >>> >> 1740903
> >>> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 45u REG 253,10
> >>> >> 65396836
> >>> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch
> (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 47u REG 253,10
> >>> >> 71229540
> >>> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch
> (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 48u REG 253,10
> >>> >> 240254078
> >>> >>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 49u REG 253,10
> >>> >> 2232423
> >>> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 51u REG 253,10
> >>> >> 96903353
> >>> >>>>>> 1210371
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 52u REG 253,10
> >>> >> 1192034
> >>> >>>>>> 1249282
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 53u REG 253,10
> >>> >> 1622135
> >>> >>>>>> 2004993
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 54u REG 253,10
> >>> >> 4172
> >>> >>>>>> 1208327
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 55u REG 253,10
> >>> >> 13496483
> >>> >>>>>> 1208325
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 56u REG 253,10
> >>> >> 12959902
> >>> >>>>>> 1208326
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 57u REG 253,10
> >>> >> 70254692
> >>> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch
> (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 58u REG 253,10
> >>> >> 581730
> >>> >>>>>> 1210373
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>>
> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 67u REG 253,10
> >>> >> 1818727
> >>> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 78u REG 253,10
> >>> >> 609905146
> >>> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 80u REG 253,10
> >>> >> 545919085
> >>> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 81u REG 253,10
> >>> >> 71032932
> >>> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch
> (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 84u REG 253,10
> >>> >> 548866111
> >>> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 85u REG 253,10
> >>> >> 564612402
> >>> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 87u REG 253,10
> >>> >> 615996179
> >>> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>> beam.smp 21352 couchdb 102u REG 253,10
> >>> >> 561111149
> >>> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> Has anyone seen this behaviour before or knows of a way around
> this
> >>> >> ?
> >>> >>>>>> Currently we have to restart the couchdb service in order to
> >>> restore
> >>> >>>> the
> >>> >>>>>> disk space.
> >>> >>>>>>
> >>> >>>>>> Any help/advice is much appreciated.
> >>> >>>>>>
> >>> >>>>>> Regards
> >>> >>>>>>
> >>> >>>>>> Stephen
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>>
> >>> >>
> >>>
> >>>
> >>
> >
>
Re: Couch not releasing deleted files
Posted by Robert Newson <ro...@gmail.com>.
It's been discussed by dev's at CouchCamp but a mailing list reminder
couldn't hurt. There's some discussion about 1.1, so perhaps that's a
good thread to raise this on.
B.
On Sun, Sep 19, 2010 at 12:24 AM, [mRg] <em...@gmail.com> wrote:
> Has this been raised with the dev list or is there a bug raised for this
> issue ? Seems a few people are having this and more are probably not seeing
> yet.
>
> On 15 September 2010 08:49, [mRg] <em...@gmail.com> wrote:
>
>> Yes there are 6 views (2 design docs)
>>
>>
>> On 14 September 2010 21:51, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>> Were there any view indexes on that DB? My suspicion is that
>>> couch_view_group misses a step somewhere in its dance and keeps a reference
>>> to an old #db{}
>>>
>>> Adam
>>>
>>> On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
>>>
>>> > It was just on a DB with a lot of writes (documents having lots of
>>> > revisions, rather that lots of new docs), the database grows as expected
>>> but
>>> > after just doing a simple compact on the db the old files still seem to
>>> be
>>> > on disk as reported previously.
>>> >
>>> > It seems its been around a while now as (as far as we can see) 0.11 /
>>> 1.0
>>> > and 1.0.1 all have the same issue, when we replicated the same data over
>>> to
>>> > the newer versions the same identical problem happened.
>>> >
>>> > On 14 September 2010 14:49, Robert Newson <ro...@gmail.com>
>>> wrote:
>>> >
>>> >> This was mentioned at CouchCamp, several developers suspect there's a
>>> >> subtle bug in the code where file descriptors are not being closed
>>> >> under some conditions.
>>> >>
>>> >> Did the OP identify a particular sequence that led here?
>>> >>
>>> >> B.
>>> >>
>>> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <
>>> juanjo@comellas.org>
>>> >> wrote:
>>> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the code,
>>> but
>>> >>> what is happening is that CouchDB is keeping file descriptors
>>> >> corresponding
>>> >>> to deleted files open, probably after compacting a database. I'm
>>> >> inferring
>>> >>> this based on the fact that with lsof I saw several cases of two files
>>> >> with
>>> >>> the same name (one deleted and one not) opened by CouchDB.
>>> >>>
>>> >>>
>>> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
>>> >>>
>>> >>>> Is this an issue with the OS or with Couch/Beam itself ?
>>> >>>>
>>> >>>> On 10 September 2010 16:57, Juan Jose Comellas <ju...@comellas.org>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only
>>> >> way
>>> >>>> I
>>> >>>>> found to solve the problem was by restarting CouchDB.
>>> >>>>>
>>> >>>>>
>>> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
>>> >>>>>
>>> >>>>>> [CouchDB Version : 0.11]
>>> >>>>>> [OS: RHEL 5.5]
>>> >>>>>>
>>> >>>>>> Hi all,
>>> >>>>>>
>>> >>>>>> We are seeing some odd behaviour with CouchDb not releasing files
>>> >> after
>>> >>>>> it
>>> >>>>>> has compacted them meaning we are seeing disk space continue to
>>> >>>> increase
>>> >>>>>> until the service is restarted even though we have regular
>>> >> compaction /
>>> >>>>>> cleanup.
>>> >>>>>>
>>> >>>>>> Typical scenario : File system alert for disk space ..
>>> >>>>>>
>>> >>>>>> df says 5.5.GB used ..
>>> >>>>>>
>>> >>>>>> df -h | grep -P 'Filesystem|db'
>>> >>>>>>
>>> >>>>>> Filesystem Size Used Avail Use% Mounted on
>>> >>>>>>
>>> >>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> .. but the files on the disk (and in Futon) say only 793MB is being
>>> >>>> used
>>> >>>>> ..
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> du -sh /var/lib/db/*
>>> >>>>>>
>>> >>>>>> 793M /var/lib/db/couchdb
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> .. running lsof shows the files marked as deleted but still there
>>> >> (I'm
>>> >>>>>> assuming beam still has some kind of handle still open to them) ..
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>>> >>>>>>
>>> >>>>>> COMMAND PID USER FD TYPE DEVICE
>>> >> SIZE
>>> >>>>>> NODE NAME
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 12u REG 253,10
>>> >> 4183
>>> >>>>>> 1210374
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 16u REG 253,10
>>> >> 612039080
>>> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 20u REG 253,10
>>> >> 180699242
>>> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 21u REG 253,10
>>> >> 3133540
>>> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 25u REG 253,10
>>> >> 71422052
>>> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 26u REG 253,10
>>> >> 71372900
>>> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 28u REG 253,10
>>> >> 621252717
>>> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 29u REG 253,10
>>> >> 1740903
>>> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 30u REG 253,10
>>> >> 3133540
>>> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 31u REG 253,10
>>> >> 2318439
>>> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 32u REG 253,10
>>> >> 983150
>>> >>>>>> 2004994
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 33u REG 253,10
>>> >> 3133540
>>> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 34u REG 253,10
>>> >> 70742116
>>> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 44u REG 253,10
>>> >> 1740903
>>> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 45u REG 253,10
>>> >> 65396836
>>> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 47u REG 253,10
>>> >> 71229540
>>> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 48u REG 253,10
>>> >> 240254078
>>> >>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 49u REG 253,10
>>> >> 2232423
>>> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 51u REG 253,10
>>> >> 96903353
>>> >>>>>> 1210371
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 52u REG 253,10
>>> >> 1192034
>>> >>>>>> 1249282
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 53u REG 253,10
>>> >> 1622135
>>> >>>>>> 2004993
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 54u REG 253,10
>>> >> 4172
>>> >>>>>> 1208327
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 55u REG 253,10
>>> >> 13496483
>>> >>>>>> 1208325
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 56u REG 253,10
>>> >> 12959902
>>> >>>>>> 1208326
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 57u REG 253,10
>>> >> 70254692
>>> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 58u REG 253,10
>>> >> 581730
>>> >>>>>> 1210373
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 67u REG 253,10
>>> >> 1818727
>>> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 78u REG 253,10
>>> >> 609905146
>>> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 80u REG 253,10
>>> >> 545919085
>>> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 81u REG 253,10
>>> >> 71032932
>>> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 84u REG 253,10
>>> >> 548866111
>>> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 85u REG 253,10
>>> >> 564612402
>>> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 87u REG 253,10
>>> >> 615996179
>>> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp 21352 couchdb 102u REG 253,10
>>> >> 561111149
>>> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Has anyone seen this behaviour before or knows of a way around this
>>> >> ?
>>> >>>>>> Currently we have to restart the couchdb service in order to
>>> restore
>>> >>>> the
>>> >>>>>> disk space.
>>> >>>>>>
>>> >>>>>> Any help/advice is much appreciated.
>>> >>>>>>
>>> >>>>>> Regards
>>> >>>>>>
>>> >>>>>> Stephen
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>>
>>>
>>
>
Re: Couch not releasing deleted files
Posted by "[mRg]" <em...@gmail.com>.
Has this been raised with the dev list or is there a bug raised for this
issue ? Seems a few people are having this and more are probably not seeing
yet.
On 15 September 2010 08:49, [mRg] <em...@gmail.com> wrote:
> Yes there are 6 views (2 design docs)
>
>
> On 14 September 2010 21:51, Adam Kocoloski <ko...@apache.org> wrote:
>
>> Were there any view indexes on that DB? My suspicion is that
>> couch_view_group misses a step somewhere in its dance and keeps a reference
>> to an old #db{}
>>
>> Adam
>>
>> On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
>>
>> > It was just on a DB with a lot of writes (documents having lots of
>> > revisions, rather that lots of new docs), the database grows as expected
>> but
>> > after just doing a simple compact on the db the old files still seem to
>> be
>> > on disk as reported previously.
>> >
>> > It seems its been around a while now as (as far as we can see) 0.11 /
>> 1.0
>> > and 1.0.1 all have the same issue, when we replicated the same data over
>> to
>> > the newer versions the same identical problem happened.
>> >
>> > On 14 September 2010 14:49, Robert Newson <ro...@gmail.com>
>> wrote:
>> >
>> >> This was mentioned at CouchCamp, several developers suspect there's a
>> >> subtle bug in the code where file descriptors are not being closed
>> >> under some conditions.
>> >>
>> >> Did the OP identify a particular sequence that led here?
>> >>
>> >> B.
>> >>
>> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <
>> juanjo@comellas.org>
>> >> wrote:
>> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the code,
>> but
>> >>> what is happening is that CouchDB is keeping file descriptors
>> >> corresponding
>> >>> to deleted files open, probably after compacting a database. I'm
>> >> inferring
>> >>> this based on the fact that with lsof I saw several cases of two files
>> >> with
>> >>> the same name (one deleted and one not) opened by CouchDB.
>> >>>
>> >>>
>> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
>> >>>
>> >>>> Is this an issue with the OS or with Couch/Beam itself ?
>> >>>>
>> >>>> On 10 September 2010 16:57, Juan Jose Comellas <ju...@comellas.org>
>> >>>> wrote:
>> >>>>
>> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only
>> >> way
>> >>>> I
>> >>>>> found to solve the problem was by restarting CouchDB.
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
>> >>>>>
>> >>>>>> [CouchDB Version : 0.11]
>> >>>>>> [OS: RHEL 5.5]
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> We are seeing some odd behaviour with CouchDb not releasing files
>> >> after
>> >>>>> it
>> >>>>>> has compacted them meaning we are seeing disk space continue to
>> >>>> increase
>> >>>>>> until the service is restarted even though we have regular
>> >> compaction /
>> >>>>>> cleanup.
>> >>>>>>
>> >>>>>> Typical scenario : File system alert for disk space ..
>> >>>>>>
>> >>>>>> df says 5.5.GB used ..
>> >>>>>>
>> >>>>>> df -h | grep -P 'Filesystem|db'
>> >>>>>>
>> >>>>>> Filesystem Size Used Avail Use% Mounted on
>> >>>>>>
>> >>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db
>> >>>>>>
>> >>>>>>
>> >>>>>> .. but the files on the disk (and in Futon) say only 793MB is being
>> >>>> used
>> >>>>> ..
>> >>>>>>
>> >>>>>>
>> >>>>>> du -sh /var/lib/db/*
>> >>>>>>
>> >>>>>> 793M /var/lib/db/couchdb
>> >>>>>>
>> >>>>>>
>> >>>>>> .. running lsof shows the files marked as deleted but still there
>> >> (I'm
>> >>>>>> assuming beam still has some kind of handle still open to them) ..
>> >>>>>>
>> >>>>>>
>> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>> >>>>>>
>> >>>>>> COMMAND PID USER FD TYPE DEVICE
>> >> SIZE
>> >>>>>> NODE NAME
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 12u REG 253,10
>> >> 4183
>> >>>>>> 1210374
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 16u REG 253,10
>> >> 612039080
>> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 20u REG 253,10
>> >> 180699242
>> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 21u REG 253,10
>> >> 3133540
>> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 25u REG 253,10
>> >> 71422052
>> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 26u REG 253,10
>> >> 71372900
>> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 28u REG 253,10
>> >> 621252717
>> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 29u REG 253,10
>> >> 1740903
>> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 30u REG 253,10
>> >> 3133540
>> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 31u REG 253,10
>> >> 2318439
>> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 32u REG 253,10
>> >> 983150
>> >>>>>> 2004994
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 33u REG 253,10
>> >> 3133540
>> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 34u REG 253,10
>> >> 70742116
>> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 44u REG 253,10
>> >> 1740903
>> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 45u REG 253,10
>> >> 65396836
>> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 47u REG 253,10
>> >> 71229540
>> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 48u REG 253,10
>> >> 240254078
>> >>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 49u REG 253,10
>> >> 2232423
>> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 51u REG 253,10
>> >> 96903353
>> >>>>>> 1210371
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 52u REG 253,10
>> >> 1192034
>> >>>>>> 1249282
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 53u REG 253,10
>> >> 1622135
>> >>>>>> 2004993
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 54u REG 253,10
>> >> 4172
>> >>>>>> 1208327
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 55u REG 253,10
>> >> 13496483
>> >>>>>> 1208325
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 56u REG 253,10
>> >> 12959902
>> >>>>>> 1208326
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 57u REG 253,10
>> >> 70254692
>> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 58u REG 253,10
>> >> 581730
>> >>>>>> 1210373
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 67u REG 253,10
>> >> 1818727
>> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 78u REG 253,10
>> >> 609905146
>> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 80u REG 253,10
>> >> 545919085
>> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 81u REG 253,10
>> >> 71032932
>> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 84u REG 253,10
>> >> 548866111
>> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 85u REG 253,10
>> >> 564612402
>> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 87u REG 253,10
>> >> 615996179
>> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>> beam.smp 21352 couchdb 102u REG 253,10
>> >> 561111149
>> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>> >>>>>>
>> >>>>>>
>> >>>>>> Has anyone seen this behaviour before or knows of a way around this
>> >> ?
>> >>>>>> Currently we have to restart the couchdb service in order to
>> restore
>> >>>> the
>> >>>>>> disk space.
>> >>>>>>
>> >>>>>> Any help/advice is much appreciated.
>> >>>>>>
>> >>>>>> Regards
>> >>>>>>
>> >>>>>> Stephen
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>>
>
Re: Couch not releasing deleted files
Posted by "[mRg]" <em...@gmail.com>.
Yes there are 6 views (2 design docs)
On 14 September 2010 21:51, Adam Kocoloski <ko...@apache.org> wrote:
> Were there any view indexes on that DB? My suspicion is that
> couch_view_group misses a step somewhere in its dance and keeps a reference
> to an old #db{}
>
> Adam
>
> On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
>
> > It was just on a DB with a lot of writes (documents having lots of
> > revisions, rather that lots of new docs), the database grows as expected
> but
> > after just doing a simple compact on the db the old files still seem to
> be
> > on disk as reported previously.
> >
> > It seems its been around a while now as (as far as we can see) 0.11 / 1.0
> > and 1.0.1 all have the same issue, when we replicated the same data over
> to
> > the newer versions the same identical problem happened.
> >
> > On 14 September 2010 14:49, Robert Newson <ro...@gmail.com>
> wrote:
> >
> >> This was mentioned at CouchCamp, several developers suspect there's a
> >> subtle bug in the code where file descriptors are not being closed
> >> under some conditions.
> >>
> >> Did the OP identify a particular sequence that led here?
> >>
> >> B.
> >>
> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <
> juanjo@comellas.org>
> >> wrote:
> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the code,
> but
> >>> what is happening is that CouchDB is keeping file descriptors
> >> corresponding
> >>> to deleted files open, probably after compacting a database. I'm
> >> inferring
> >>> this based on the fact that with lsof I saw several cases of two files
> >> with
> >>> the same name (one deleted and one not) opened by CouchDB.
> >>>
> >>>
> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
> >>>
> >>>> Is this an issue with the OS or with Couch/Beam itself ?
> >>>>
> >>>> On 10 September 2010 16:57, Juan Jose Comellas <ju...@comellas.org>
> >>>> wrote:
> >>>>
> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only
> >> way
> >>>> I
> >>>>> found to solve the problem was by restarting CouchDB.
> >>>>>
> >>>>>
> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
> >>>>>
> >>>>>> [CouchDB Version : 0.11]
> >>>>>> [OS: RHEL 5.5]
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> We are seeing some odd behaviour with CouchDb not releasing files
> >> after
> >>>>> it
> >>>>>> has compacted them meaning we are seeing disk space continue to
> >>>> increase
> >>>>>> until the service is restarted even though we have regular
> >> compaction /
> >>>>>> cleanup.
> >>>>>>
> >>>>>> Typical scenario : File system alert for disk space ..
> >>>>>>
> >>>>>> df says 5.5.GB used ..
> >>>>>>
> >>>>>> df -h | grep -P 'Filesystem|db'
> >>>>>>
> >>>>>> Filesystem Size Used Avail Use% Mounted on
> >>>>>>
> >>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db
> >>>>>>
> >>>>>>
> >>>>>> .. but the files on the disk (and in Futon) say only 793MB is being
> >>>> used
> >>>>> ..
> >>>>>>
> >>>>>>
> >>>>>> du -sh /var/lib/db/*
> >>>>>>
> >>>>>> 793M /var/lib/db/couchdb
> >>>>>>
> >>>>>>
> >>>>>> .. running lsof shows the files marked as deleted but still there
> >> (I'm
> >>>>>> assuming beam still has some kind of handle still open to them) ..
> >>>>>>
> >>>>>>
> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
> >>>>>>
> >>>>>> COMMAND PID USER FD TYPE DEVICE
> >> SIZE
> >>>>>> NODE NAME
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 12u REG 253,10
> >> 4183
> >>>>>> 1210374
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 16u REG 253,10
> >> 612039080
> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 20u REG 253,10
> >> 180699242
> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 21u REG 253,10
> >> 3133540
> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 25u REG 253,10
> >> 71422052
> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 26u REG 253,10
> >> 71372900
> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 28u REG 253,10
> >> 621252717
> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 29u REG 253,10
> >> 1740903
> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 30u REG 253,10
> >> 3133540
> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 31u REG 253,10
> >> 2318439
> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 32u REG 253,10
> >> 983150
> >>>>>> 2004994
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 33u REG 253,10
> >> 3133540
> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 34u REG 253,10
> >> 70742116
> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 44u REG 253,10
> >> 1740903
> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 45u REG 253,10
> >> 65396836
> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 47u REG 253,10
> >> 71229540
> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 48u REG 253,10
> >> 240254078
> >>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 49u REG 253,10
> >> 2232423
> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 51u REG 253,10
> >> 96903353
> >>>>>> 1210371
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 52u REG 253,10
> >> 1192034
> >>>>>> 1249282
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 53u REG 253,10
> >> 1622135
> >>>>>> 2004993
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 54u REG 253,10
> >> 4172
> >>>>>> 1208327
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 55u REG 253,10
> >> 13496483
> >>>>>> 1208325
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 56u REG 253,10
> >> 12959902
> >>>>>> 1208326
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 57u REG 253,10
> >> 70254692
> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 58u REG 253,10
> >> 581730
> >>>>>> 1210373
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 67u REG 253,10
> >> 1818727
> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 78u REG 253,10
> >> 609905146
> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 80u REG 253,10
> >> 545919085
> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 81u REG 253,10
> >> 71032932
> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 84u REG 253,10
> >> 548866111
> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 85u REG 253,10
> >> 564612402
> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 87u REG 253,10
> >> 615996179
> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>> beam.smp 21352 couchdb 102u REG 253,10
> >> 561111149
> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
> >>>>>>
> >>>>>>
> >>>>>> Has anyone seen this behaviour before or knows of a way around this
> >> ?
> >>>>>> Currently we have to restart the couchdb service in order to restore
> >>>> the
> >>>>>> disk space.
> >>>>>>
> >>>>>> Any help/advice is much appreciated.
> >>>>>>
> >>>>>> Regards
> >>>>>>
> >>>>>> Stephen
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: Couch not releasing deleted files
Posted by Adam Kocoloski <ko...@apache.org>.
Were there any view indexes on that DB? My suspicion is that couch_view_group misses a step somewhere in its dance and keeps a reference to an old #db{}
Adam
On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
> It was just on a DB with a lot of writes (documents having lots of
> revisions, rather that lots of new docs), the database grows as expected but
> after just doing a simple compact on the db the old files still seem to be
> on disk as reported previously.
>
> It seems its been around a while now as (as far as we can see) 0.11 / 1.0
> and 1.0.1 all have the same issue, when we replicated the same data over to
> the newer versions the same identical problem happened.
>
> On 14 September 2010 14:49, Robert Newson <ro...@gmail.com> wrote:
>
>> This was mentioned at CouchCamp, several developers suspect there's a
>> subtle bug in the code where file descriptors are not being closed
>> under some conditions.
>>
>> Did the OP identify a particular sequence that led here?
>>
>> B.
>>
>> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <ju...@comellas.org>
>> wrote:
>>> I'm pretty certain it's a bug in CouchDB. I haven't checked the code, but
>>> what is happening is that CouchDB is keeping file descriptors
>> corresponding
>>> to deleted files open, probably after compacting a database. I'm
>> inferring
>>> this based on the fact that with lsof I saw several cases of two files
>> with
>>> the same name (one deleted and one not) opened by CouchDB.
>>>
>>>
>>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
>>>
>>>> Is this an issue with the OS or with Couch/Beam itself ?
>>>>
>>>> On 10 September 2010 16:57, Juan Jose Comellas <ju...@comellas.org>
>>>> wrote:
>>>>
>>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only
>> way
>>>> I
>>>>> found to solve the problem was by restarting CouchDB.
>>>>>
>>>>>
>>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
>>>>>
>>>>>> [CouchDB Version : 0.11]
>>>>>> [OS: RHEL 5.5]
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We are seeing some odd behaviour with CouchDb not releasing files
>> after
>>>>> it
>>>>>> has compacted them meaning we are seeing disk space continue to
>>>> increase
>>>>>> until the service is restarted even though we have regular
>> compaction /
>>>>>> cleanup.
>>>>>>
>>>>>> Typical scenario : File system alert for disk space ..
>>>>>>
>>>>>> df says 5.5.GB used ..
>>>>>>
>>>>>> df -h | grep -P 'Filesystem|db'
>>>>>>
>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>
>>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db
>>>>>>
>>>>>>
>>>>>> .. but the files on the disk (and in Futon) say only 793MB is being
>>>> used
>>>>> ..
>>>>>>
>>>>>>
>>>>>> du -sh /var/lib/db/*
>>>>>>
>>>>>> 793M /var/lib/db/couchdb
>>>>>>
>>>>>>
>>>>>> .. running lsof shows the files marked as deleted but still there
>> (I'm
>>>>>> assuming beam still has some kind of handle still open to them) ..
>>>>>>
>>>>>>
>>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>>>>>>
>>>>>> COMMAND PID USER FD TYPE DEVICE
>> SIZE
>>>>>> NODE NAME
>>>>>>
>>>>>> beam.smp 21352 couchdb 12u REG 253,10
>> 4183
>>>>>> 1210374
>>>>>>
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 16u REG 253,10
>> 612039080
>>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 20u REG 253,10
>> 180699242
>>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 21u REG 253,10
>> 3133540
>>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 25u REG 253,10
>> 71422052
>>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 26u REG 253,10
>> 71372900
>>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 28u REG 253,10
>> 621252717
>>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 29u REG 253,10
>> 1740903
>>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 30u REG 253,10
>> 3133540
>>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 31u REG 253,10
>> 2318439
>>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 32u REG 253,10
>> 983150
>>>>>> 2004994
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 33u REG 253,10
>> 3133540
>>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 34u REG 253,10
>> 70742116
>>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 44u REG 253,10
>> 1740903
>>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 45u REG 253,10
>> 65396836
>>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 47u REG 253,10
>> 71229540
>>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 48u REG 253,10
>> 240254078
>>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 49u REG 253,10
>> 2232423
>>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 51u REG 253,10
>> 96903353
>>>>>> 1210371
>>>>>>
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 52u REG 253,10
>> 1192034
>>>>>> 1249282
>>>>>>
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 53u REG 253,10
>> 1622135
>>>>>> 2004993
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 54u REG 253,10
>> 4172
>>>>>> 1208327
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 55u REG 253,10
>> 13496483
>>>>>> 1208325
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 56u REG 253,10
>> 12959902
>>>>>> 1208326
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 57u REG 253,10
>> 70254692
>>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 58u REG 253,10
>> 581730
>>>>>> 1210373
>>>>>>
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 67u REG 253,10
>> 1818727
>>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 78u REG 253,10
>> 609905146
>>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 80u REG 253,10
>> 545919085
>>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 81u REG 253,10
>> 71032932
>>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 84u REG 253,10
>> 548866111
>>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 85u REG 253,10
>> 564612402
>>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 87u REG 253,10
>> 615996179
>>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 102u REG 253,10
>> 561111149
>>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>>
>>>>>> Has anyone seen this behaviour before or knows of a way around this
>> ?
>>>>>> Currently we have to restart the couchdb service in order to restore
>>>> the
>>>>>> disk space.
>>>>>>
>>>>>> Any help/advice is much appreciated.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Stephen
>>>>>>
>>>>>
>>>>
>>>
>>
Re: Couch not releasing deleted files
Posted by "[mRg]" <em...@gmail.com>.
It was just on a DB with a lot of writes (documents having lots of
revisions, rather that lots of new docs), the database grows as expected but
after just doing a simple compact on the db the old files still seem to be
on disk as reported previously.
It seems its been around a while now as (as far as we can see) 0.11 / 1.0
and 1.0.1 all have the same issue, when we replicated the same data over to
the newer versions the same identical problem happened.
On 14 September 2010 14:49, Robert Newson <ro...@gmail.com> wrote:
> This was mentioned at CouchCamp, several developers suspect there's a
> subtle bug in the code where file descriptors are not being closed
> under some conditions.
>
> Did the OP identify a particular sequence that led here?
>
> B.
>
> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <ju...@comellas.org>
> wrote:
> > I'm pretty certain it's a bug in CouchDB. I haven't checked the code, but
> > what is happening is that CouchDB is keeping file descriptors
> corresponding
> > to deleted files open, probably after compacting a database. I'm
> inferring
> > this based on the fact that with lsof I saw several cases of two files
> with
> > the same name (one deleted and one not) opened by CouchDB.
> >
> >
> > On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
> >
> >> Is this an issue with the OS or with Couch/Beam itself ?
> >>
> >> On 10 September 2010 16:57, Juan Jose Comellas <ju...@comellas.org>
> >> wrote:
> >>
> >> > It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only
> way
> >> I
> >> > found to solve the problem was by restarting CouchDB.
> >> >
> >> >
> >> > On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
> >> >
> >> > > [CouchDB Version : 0.11]
> >> > > [OS: RHEL 5.5]
> >> > >
> >> > > Hi all,
> >> > >
> >> > > We are seeing some odd behaviour with CouchDb not releasing files
> after
> >> > it
> >> > > has compacted them meaning we are seeing disk space continue to
> >> increase
> >> > > until the service is restarted even though we have regular
> compaction /
> >> > > cleanup.
> >> > >
> >> > > Typical scenario : File system alert for disk space ..
> >> > >
> >> > > df says 5.5.GB used ..
> >> > >
> >> > > df -h | grep -P 'Filesystem|db'
> >> > >
> >> > > Filesystem Size Used Avail Use% Mounted on
> >> > >
> >> > > 8.0G 5.5G 2.2G 72% /var/lib/db
> >> > >
> >> > >
> >> > > .. but the files on the disk (and in Futon) say only 793MB is being
> >> used
> >> > ..
> >> > >
> >> > >
> >> > > du -sh /var/lib/db/*
> >> > >
> >> > > 793M /var/lib/db/couchdb
> >> > >
> >> > >
> >> > > .. running lsof shows the files marked as deleted but still there
> (I'm
> >> > > assuming beam still has some kind of handle still open to them) ..
> >> > >
> >> > >
> >> > > lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
> >> > >
> >> > > COMMAND PID USER FD TYPE DEVICE
> SIZE
> >> > > NODE NAME
> >> > >
> >> > > beam.smp 21352 couchdb 12u REG 253,10
> 4183
> >> > > 1210374
> >> > >
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
> >> > >
> >> > > beam.smp 21352 couchdb 16u REG 253,10
> 612039080
> >> > > 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 20u REG 253,10
> 180699242
> >> > > 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 21u REG 253,10
> 3133540
> >> > > 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 25u REG 253,10
> 71422052
> >> > > 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 26u REG 253,10
> 71372900
> >> > > 2056193 /var/lib/db/couchdb/active_context_schedule.couch
> >> > >
> >> > > beam.smp 21352 couchdb 28u REG 253,10
> 621252717
> >> > > 1204238 /var/lib/db/couchdb/active_context.couch
> >> > >
> >> > > beam.smp 21352 couchdb 29u REG 253,10
> 1740903
> >> > > 90113 /var/lib/db/couchdb/mapping_cache.couch
> >> > >
> >> > > beam.smp 21352 couchdb 30u REG 253,10
> 3133540
> >> > > 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 31u REG 253,10
> 2318439
> >> > > 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 32u REG 253,10
> 983150
> >> > > 2004994
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
> >> > >
> >> > > beam.smp 21352 couchdb 33u REG 253,10
> 3133540
> >> > > 2056194 /var/lib/db/couchdb/thesaurus.couch
> >> > >
> >> > > beam.smp 21352 couchdb 34u REG 253,10
> 70742116
> >> > > 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 44u REG 253,10
> 1740903
> >> > > 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 45u REG 253,10
> 65396836
> >> > > 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 47u REG 253,10
> 71229540
> >> > > 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 48u REG 253,10
> 240254078
> >> > > 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 49u REG 253,10
> 2232423
> >> > > 1204241 /var/lib/db/couchdb/mappings.couch
> >> > >
> >> > > beam.smp 21352 couchdb 51u REG 253,10
> 96903353
> >> > > 1210371
> >> > >
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
> >> > >
> >> > > beam.smp 21352 couchdb 52u REG 253,10
> 1192034
> >> > > 1249282
> >> > >
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
> >> > >
> >> > > beam.smp 21352 couchdb 53u REG 253,10
> 1622135
> >> > > 2004993
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
> >> > >
> >> > > beam.smp 21352 couchdb 54u REG 253,10
> 4172
> >> > > 1208327
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
> >> > >
> >> > > beam.smp 21352 couchdb 55u REG 253,10
> 13496483
> >> > > 1208325
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
> >> > >
> >> > > beam.smp 21352 couchdb 56u REG 253,10
> 12959902
> >> > > 1208326
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
> >> > >
> >> > > beam.smp 21352 couchdb 57u REG 253,10
> 70254692
> >> > > 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 58u REG 253,10
> 581730
> >> > > 1210373
> >> > >
> >> > >
> >> >
> >>
> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
> >> > >
> >> > > beam.smp 21352 couchdb 67u REG 253,10
> 1818727
> >> > > 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 78u REG 253,10
> 609905146
> >> > > 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 80u REG 253,10
> 545919085
> >> > > 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 81u REG 253,10
> 71032932
> >> > > 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 84u REG 253,10
> 548866111
> >> > > 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 85u REG 253,10
> 564612402
> >> > > 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 87u REG 253,10
> 615996179
> >> > > 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > > beam.smp 21352 couchdb 102u REG 253,10
> 561111149
> >> > > 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
> >> > >
> >> > >
> >> > > Has anyone seen this behaviour before or knows of a way around this
> ?
> >> > > Currently we have to restart the couchdb service in order to restore
> >> the
> >> > > disk space.
> >> > >
> >> > > Any help/advice is much appreciated.
> >> > >
> >> > > Regards
> >> > >
> >> > > Stephen
> >> > >
> >> >
> >>
> >
>
Re: Couch not releasing deleted files
Posted by Robert Newson <ro...@gmail.com>.
This was mentioned at CouchCamp, several developers suspect there's a
subtle bug in the code where file descriptors are not being closed
under some conditions.
Did the OP identify a particular sequence that led here?
B.
On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <ju...@comellas.org> wrote:
> I'm pretty certain it's a bug in CouchDB. I haven't checked the code, but
> what is happening is that CouchDB is keeping file descriptors corresponding
> to deleted files open, probably after compacting a database. I'm inferring
> this based on the fact that with lsof I saw several cases of two files with
> the same name (one deleted and one not) opened by CouchDB.
>
>
> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
>
>> Is this an issue with the OS or with Couch/Beam itself ?
>>
>> On 10 September 2010 16:57, Juan Jose Comellas <ju...@comellas.org>
>> wrote:
>>
>> > It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only way
>> I
>> > found to solve the problem was by restarting CouchDB.
>> >
>> >
>> > On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
>> >
>> > > [CouchDB Version : 0.11]
>> > > [OS: RHEL 5.5]
>> > >
>> > > Hi all,
>> > >
>> > > We are seeing some odd behaviour with CouchDb not releasing files after
>> > it
>> > > has compacted them meaning we are seeing disk space continue to
>> increase
>> > > until the service is restarted even though we have regular compaction /
>> > > cleanup.
>> > >
>> > > Typical scenario : File system alert for disk space ..
>> > >
>> > > df says 5.5.GB used ..
>> > >
>> > > df -h | grep -P 'Filesystem|db'
>> > >
>> > > Filesystem Size Used Avail Use% Mounted on
>> > >
>> > > 8.0G 5.5G 2.2G 72% /var/lib/db
>> > >
>> > >
>> > > .. but the files on the disk (and in Futon) say only 793MB is being
>> used
>> > ..
>> > >
>> > >
>> > > du -sh /var/lib/db/*
>> > >
>> > > 793M /var/lib/db/couchdb
>> > >
>> > >
>> > > .. running lsof shows the files marked as deleted but still there (I'm
>> > > assuming beam still has some kind of handle still open to them) ..
>> > >
>> > >
>> > > lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>> > >
>> > > COMMAND PID USER FD TYPE DEVICE SIZE
>> > > NODE NAME
>> > >
>> > > beam.smp 21352 couchdb 12u REG 253,10 4183
>> > > 1210374
>> > >
>> > >
>> >
>> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>> > >
>> > > beam.smp 21352 couchdb 16u REG 253,10 612039080
>> > > 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 20u REG 253,10 180699242
>> > > 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 21u REG 253,10 3133540
>> > > 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 25u REG 253,10 71422052
>> > > 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 26u REG 253,10 71372900
>> > > 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>> > >
>> > > beam.smp 21352 couchdb 28u REG 253,10 621252717
>> > > 1204238 /var/lib/db/couchdb/active_context.couch
>> > >
>> > > beam.smp 21352 couchdb 29u REG 253,10 1740903
>> > > 90113 /var/lib/db/couchdb/mapping_cache.couch
>> > >
>> > > beam.smp 21352 couchdb 30u REG 253,10 3133540
>> > > 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 31u REG 253,10 2318439
>> > > 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 32u REG 253,10 983150
>> > > 2004994
>> > >
>> >
>> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>> > >
>> > > beam.smp 21352 couchdb 33u REG 253,10 3133540
>> > > 2056194 /var/lib/db/couchdb/thesaurus.couch
>> > >
>> > > beam.smp 21352 couchdb 34u REG 253,10 70742116
>> > > 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 44u REG 253,10 1740903
>> > > 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 45u REG 253,10 65396836
>> > > 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 47u REG 253,10 71229540
>> > > 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 48u REG 253,10 240254078
>> > > 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 49u REG 253,10 2232423
>> > > 1204241 /var/lib/db/couchdb/mappings.couch
>> > >
>> > > beam.smp 21352 couchdb 51u REG 253,10 96903353
>> > > 1210371
>> > >
>> > >
>> >
>> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>> > >
>> > > beam.smp 21352 couchdb 52u REG 253,10 1192034
>> > > 1249282
>> > >
>> > >
>> >
>> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>> > >
>> > > beam.smp 21352 couchdb 53u REG 253,10 1622135
>> > > 2004993
>> > >
>> >
>> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>> > >
>> > > beam.smp 21352 couchdb 54u REG 253,10 4172
>> > > 1208327
>> > >
>> >
>> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>> > >
>> > > beam.smp 21352 couchdb 55u REG 253,10 13496483
>> > > 1208325
>> > >
>> >
>> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>> > >
>> > > beam.smp 21352 couchdb 56u REG 253,10 12959902
>> > > 1208326
>> > >
>> >
>> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>> > >
>> > > beam.smp 21352 couchdb 57u REG 253,10 70254692
>> > > 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 58u REG 253,10 581730
>> > > 1210373
>> > >
>> > >
>> >
>> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>> > >
>> > > beam.smp 21352 couchdb 67u REG 253,10 1818727
>> > > 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 78u REG 253,10 609905146
>> > > 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 80u REG 253,10 545919085
>> > > 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 81u REG 253,10 71032932
>> > > 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 84u REG 253,10 548866111
>> > > 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 85u REG 253,10 564612402
>> > > 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 87u REG 253,10 615996179
>> > > 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp 21352 couchdb 102u REG 253,10 561111149
>> > > 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > >
>> > > Has anyone seen this behaviour before or knows of a way around this ?
>> > > Currently we have to restart the couchdb service in order to restore
>> the
>> > > disk space.
>> > >
>> > > Any help/advice is much appreciated.
>> > >
>> > > Regards
>> > >
>> > > Stephen
>> > >
>> >
>>
>
Re: Couch not releasing deleted files
Posted by Juan Jose Comellas <ju...@comellas.org>.
I'm pretty certain it's a bug in CouchDB. I haven't checked the code, but
what is happening is that CouchDB is keeping file descriptors corresponding
to deleted files open, probably after compacting a database. I'm inferring
this based on the fact that with lsof I saw several cases of two files with
the same name (one deleted and one not) opened by CouchDB.
On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <em...@gmail.com> wrote:
> Is this an issue with the OS or with Couch/Beam itself ?
>
> On 10 September 2010 16:57, Juan Jose Comellas <ju...@comellas.org>
> wrote:
>
> > It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only way
> I
> > found to solve the problem was by restarting CouchDB.
> >
> >
> > On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
> >
> > > [CouchDB Version : 0.11]
> > > [OS: RHEL 5.5]
> > >
> > > Hi all,
> > >
> > > We are seeing some odd behaviour with CouchDb not releasing files after
> > it
> > > has compacted them meaning we are seeing disk space continue to
> increase
> > > until the service is restarted even though we have regular compaction /
> > > cleanup.
> > >
> > > Typical scenario : File system alert for disk space ..
> > >
> > > df says 5.5.GB used ..
> > >
> > > df -h | grep -P 'Filesystem|db'
> > >
> > > Filesystem Size Used Avail Use% Mounted on
> > >
> > > 8.0G 5.5G 2.2G 72% /var/lib/db
> > >
> > >
> > > .. but the files on the disk (and in Futon) say only 793MB is being
> used
> > ..
> > >
> > >
> > > du -sh /var/lib/db/*
> > >
> > > 793M /var/lib/db/couchdb
> > >
> > >
> > > .. running lsof shows the files marked as deleted but still there (I'm
> > > assuming beam still has some kind of handle still open to them) ..
> > >
> > >
> > > lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
> > >
> > > COMMAND PID USER FD TYPE DEVICE SIZE
> > > NODE NAME
> > >
> > > beam.smp 21352 couchdb 12u REG 253,10 4183
> > > 1210374
> > >
> > >
> >
> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
> > >
> > > beam.smp 21352 couchdb 16u REG 253,10 612039080
> > > 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 20u REG 253,10 180699242
> > > 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 21u REG 253,10 3133540
> > > 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 25u REG 253,10 71422052
> > > 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 26u REG 253,10 71372900
> > > 2056193 /var/lib/db/couchdb/active_context_schedule.couch
> > >
> > > beam.smp 21352 couchdb 28u REG 253,10 621252717
> > > 1204238 /var/lib/db/couchdb/active_context.couch
> > >
> > > beam.smp 21352 couchdb 29u REG 253,10 1740903
> > > 90113 /var/lib/db/couchdb/mapping_cache.couch
> > >
> > > beam.smp 21352 couchdb 30u REG 253,10 3133540
> > > 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 31u REG 253,10 2318439
> > > 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 32u REG 253,10 983150
> > > 2004994
> > >
> >
> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
> > >
> > > beam.smp 21352 couchdb 33u REG 253,10 3133540
> > > 2056194 /var/lib/db/couchdb/thesaurus.couch
> > >
> > > beam.smp 21352 couchdb 34u REG 253,10 70742116
> > > 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 44u REG 253,10 1740903
> > > 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 45u REG 253,10 65396836
> > > 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 47u REG 253,10 71229540
> > > 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 48u REG 253,10 240254078
> > > 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 49u REG 253,10 2232423
> > > 1204241 /var/lib/db/couchdb/mappings.couch
> > >
> > > beam.smp 21352 couchdb 51u REG 253,10 96903353
> > > 1210371
> > >
> > >
> >
> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
> > >
> > > beam.smp 21352 couchdb 52u REG 253,10 1192034
> > > 1249282
> > >
> > >
> >
> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
> > >
> > > beam.smp 21352 couchdb 53u REG 253,10 1622135
> > > 2004993
> > >
> >
> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
> > >
> > > beam.smp 21352 couchdb 54u REG 253,10 4172
> > > 1208327
> > >
> >
> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
> > >
> > > beam.smp 21352 couchdb 55u REG 253,10 13496483
> > > 1208325
> > >
> >
> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
> > >
> > > beam.smp 21352 couchdb 56u REG 253,10 12959902
> > > 1208326
> > >
> >
> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
> > >
> > > beam.smp 21352 couchdb 57u REG 253,10 70254692
> > > 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 58u REG 253,10 581730
> > > 1210373
> > >
> > >
> >
> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
> > >
> > > beam.smp 21352 couchdb 67u REG 253,10 1818727
> > > 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 78u REG 253,10 609905146
> > > 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 80u REG 253,10 545919085
> > > 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 81u REG 253,10 71032932
> > > 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 84u REG 253,10 548866111
> > > 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 85u REG 253,10 564612402
> > > 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 87u REG 253,10 615996179
> > > 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > > beam.smp 21352 couchdb 102u REG 253,10 561111149
> > > 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
> > >
> > >
> > > Has anyone seen this behaviour before or knows of a way around this ?
> > > Currently we have to restart the couchdb service in order to restore
> the
> > > disk space.
> > >
> > > Any help/advice is much appreciated.
> > >
> > > Regards
> > >
> > > Stephen
> > >
> >
>
Re: Couch not releasing deleted files
Posted by "[mRg]" <em...@gmail.com>.
Is this an issue with the OS or with Couch/Beam itself ?
On 10 September 2010 16:57, Juan Jose Comellas <ju...@comellas.org> wrote:
> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only way I
> found to solve the problem was by restarting CouchDB.
>
>
> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
>
> > [CouchDB Version : 0.11]
> > [OS: RHEL 5.5]
> >
> > Hi all,
> >
> > We are seeing some odd behaviour with CouchDb not releasing files after
> it
> > has compacted them meaning we are seeing disk space continue to increase
> > until the service is restarted even though we have regular compaction /
> > cleanup.
> >
> > Typical scenario : File system alert for disk space ..
> >
> > df says 5.5.GB used ..
> >
> > df -h | grep -P 'Filesystem|db'
> >
> > Filesystem Size Used Avail Use% Mounted on
> >
> > 8.0G 5.5G 2.2G 72% /var/lib/db
> >
> >
> > .. but the files on the disk (and in Futon) say only 793MB is being used
> ..
> >
> >
> > du -sh /var/lib/db/*
> >
> > 793M /var/lib/db/couchdb
> >
> >
> > .. running lsof shows the files marked as deleted but still there (I'm
> > assuming beam still has some kind of handle still open to them) ..
> >
> >
> > lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
> >
> > COMMAND PID USER FD TYPE DEVICE SIZE
> > NODE NAME
> >
> > beam.smp 21352 couchdb 12u REG 253,10 4183
> > 1210374
> >
> >
> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
> >
> > beam.smp 21352 couchdb 16u REG 253,10 612039080
> > 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> > beam.smp 21352 couchdb 20u REG 253,10 180699242
> > 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> > beam.smp 21352 couchdb 21u REG 253,10 3133540
> > 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
> >
> > beam.smp 21352 couchdb 25u REG 253,10 71422052
> > 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >
> > beam.smp 21352 couchdb 26u REG 253,10 71372900
> > 2056193 /var/lib/db/couchdb/active_context_schedule.couch
> >
> > beam.smp 21352 couchdb 28u REG 253,10 621252717
> > 1204238 /var/lib/db/couchdb/active_context.couch
> >
> > beam.smp 21352 couchdb 29u REG 253,10 1740903
> > 90113 /var/lib/db/couchdb/mapping_cache.couch
> >
> > beam.smp 21352 couchdb 30u REG 253,10 3133540
> > 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
> >
> > beam.smp 21352 couchdb 31u REG 253,10 2318439
> > 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
> >
> > beam.smp 21352 couchdb 32u REG 253,10 983150
> > 2004994
> >
> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
> >
> > beam.smp 21352 couchdb 33u REG 253,10 3133540
> > 2056194 /var/lib/db/couchdb/thesaurus.couch
> >
> > beam.smp 21352 couchdb 34u REG 253,10 70742116
> > 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >
> > beam.smp 21352 couchdb 44u REG 253,10 1740903
> > 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> >
> > beam.smp 21352 couchdb 45u REG 253,10 65396836
> > 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >
> > beam.smp 21352 couchdb 47u REG 253,10 71229540
> > 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >
> > beam.smp 21352 couchdb 48u REG 253,10 240254078
> > 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> > beam.smp 21352 couchdb 49u REG 253,10 2232423
> > 1204241 /var/lib/db/couchdb/mappings.couch
> >
> > beam.smp 21352 couchdb 51u REG 253,10 96903353
> > 1210371
> >
> >
> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
> >
> > beam.smp 21352 couchdb 52u REG 253,10 1192034
> > 1249282
> >
> >
> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
> >
> > beam.smp 21352 couchdb 53u REG 253,10 1622135
> > 2004993
> >
> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
> >
> > beam.smp 21352 couchdb 54u REG 253,10 4172
> > 1208327
> >
> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
> >
> > beam.smp 21352 couchdb 55u REG 253,10 13496483
> > 1208325
> >
> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
> >
> > beam.smp 21352 couchdb 56u REG 253,10 12959902
> > 1208326
> >
> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
> >
> > beam.smp 21352 couchdb 57u REG 253,10 70254692
> > 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >
> > beam.smp 21352 couchdb 58u REG 253,10 581730
> > 1210373
> >
> >
> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
> >
> > beam.smp 21352 couchdb 67u REG 253,10 1818727
> > 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
> >
> > beam.smp 21352 couchdb 78u REG 253,10 609905146
> > 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> > beam.smp 21352 couchdb 80u REG 253,10 545919085
> > 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> > beam.smp 21352 couchdb 81u REG 253,10 71032932
> > 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
> >
> > beam.smp 21352 couchdb 84u REG 253,10 548866111
> > 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> > beam.smp 21352 couchdb 85u REG 253,10 564612402
> > 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> > beam.smp 21352 couchdb 87u REG 253,10 615996179
> > 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> > beam.smp 21352 couchdb 102u REG 253,10 561111149
> > 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
> >
> >
> > Has anyone seen this behaviour before or knows of a way around this ?
> > Currently we have to restart the couchdb service in order to restore the
> > disk space.
> >
> > Any help/advice is much appreciated.
> >
> > Regards
> >
> > Stephen
> >
>
Re: Couch not releasing deleted files
Posted by Juan Jose Comellas <ju...@comellas.org>.
It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only way I
found to solve the problem was by restarting CouchDB.
On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <em...@gmail.com> wrote:
> [CouchDB Version : 0.11]
> [OS: RHEL 5.5]
>
> Hi all,
>
> We are seeing some odd behaviour with CouchDb not releasing files after it
> has compacted them meaning we are seeing disk space continue to increase
> until the service is restarted even though we have regular compaction /
> cleanup.
>
> Typical scenario : File system alert for disk space ..
>
> df says 5.5.GB used ..
>
> df -h | grep -P 'Filesystem|db'
>
> Filesystem Size Used Avail Use% Mounted on
>
> 8.0G 5.5G 2.2G 72% /var/lib/db
>
>
> .. but the files on the disk (and in Futon) say only 793MB is being used ..
>
>
> du -sh /var/lib/db/*
>
> 793M /var/lib/db/couchdb
>
>
> .. running lsof shows the files marked as deleted but still there (I'm
> assuming beam still has some kind of handle still open to them) ..
>
>
> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>
> COMMAND PID USER FD TYPE DEVICE SIZE
> NODE NAME
>
> beam.smp 21352 couchdb 12u REG 253,10 4183
> 1210374
>
> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>
> beam.smp 21352 couchdb 16u REG 253,10 612039080
> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 20u REG 253,10 180699242
> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 21u REG 253,10 3133540
> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>
> beam.smp 21352 couchdb 25u REG 253,10 71422052
> 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 26u REG 253,10 71372900
> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>
> beam.smp 21352 couchdb 28u REG 253,10 621252717
> 1204238 /var/lib/db/couchdb/active_context.couch
>
> beam.smp 21352 couchdb 29u REG 253,10 1740903
> 90113 /var/lib/db/couchdb/mapping_cache.couch
>
> beam.smp 21352 couchdb 30u REG 253,10 3133540
> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>
> beam.smp 21352 couchdb 31u REG 253,10 2318439
> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>
> beam.smp 21352 couchdb 32u REG 253,10 983150
> 2004994
> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>
> beam.smp 21352 couchdb 33u REG 253,10 3133540
> 2056194 /var/lib/db/couchdb/thesaurus.couch
>
> beam.smp 21352 couchdb 34u REG 253,10 70742116
> 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 44u REG 253,10 1740903
> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>
> beam.smp 21352 couchdb 45u REG 253,10 65396836
> 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 47u REG 253,10 71229540
> 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 48u REG 253,10 240254078
> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 49u REG 253,10 2232423
> 1204241 /var/lib/db/couchdb/mappings.couch
>
> beam.smp 21352 couchdb 51u REG 253,10 96903353
> 1210371
>
> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>
> beam.smp 21352 couchdb 52u REG 253,10 1192034
> 1249282
>
> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>
> beam.smp 21352 couchdb 53u REG 253,10 1622135
> 2004993
> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>
> beam.smp 21352 couchdb 54u REG 253,10 4172
> 1208327
> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>
> beam.smp 21352 couchdb 55u REG 253,10 13496483
> 1208325
> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>
> beam.smp 21352 couchdb 56u REG 253,10 12959902
> 1208326
> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>
> beam.smp 21352 couchdb 57u REG 253,10 70254692
> 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 58u REG 253,10 581730
> 1210373
>
> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>
> beam.smp 21352 couchdb 67u REG 253,10 1818727
> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>
> beam.smp 21352 couchdb 78u REG 253,10 609905146
> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 80u REG 253,10 545919085
> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 81u REG 253,10 71032932
> 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 84u REG 253,10 548866111
> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 85u REG 253,10 564612402
> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 87u REG 253,10 615996179
> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 102u REG 253,10 561111149
> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>
>
> Has anyone seen this behaviour before or knows of a way around this ?
> Currently we have to restart the couchdb service in order to restore the
> disk space.
>
> Any help/advice is much appreciated.
>
> Regards
>
> Stephen
>
Re: Couch not releasing deleted files
Posted by "[mRg]" <em...@gmail.com>.
I also tested by replicating to a CouchDB 1.0.1 instance and the same thing
occurs (although now the unreleased files are in the .deleted directory).
I can reproduce this by the following steps:
1> Load a couchdb instance with dummy data
2> Execute a view
3> Run a compact / cleanup on the view
4> Check lsof on the .deleted directory
While 'du' reports its is gone 'df' shows the disk still in use. I thought
it may be because of some delay in cleaning the deleted nodes but a day or 2
later that are still there.
5> Restart couchdb and the files are removed.
@Panop - The only way I've found is to restart the couchdb service.
Regards
Stephen
On 10 September 2010 10:00, Panop Suvaphrom <PS...@vizrt.com> wrote:
> Yeah, me too. On couch 1.0.1 with ubuntu 10.04
> Do you have any idea to operate on this ?
>
>
>
>
> -----Original Message-----
> From: [mRg] [mailto:emargee@gmail.com]
> Sent: Thursday, September 09, 2010 8:25 PM
> To: user@couchdb.apache.org
> Subject: Couch not releasing deleted files
>
> [CouchDB Version : 0.11]
> [OS: RHEL 5.5]
>
> Hi all,
>
> We are seeing some odd behaviour with CouchDb not releasing files after it
> has compacted them meaning we are seeing disk space continue to increase
> until the service is restarted even though we have regular compaction /
> cleanup.
>
> Typical scenario : File system alert for disk space ..
>
> df says 5.5.GB used ..
>
> df -h | grep -P 'Filesystem|db'
>
> Filesystem Size Used Avail Use% Mounted on
>
> 8.0G 5.5G 2.2G 72% /var/lib/db
>
>
> .. but the files on the disk (and in Futon) say only 793MB is being used ..
>
>
> du -sh /var/lib/db/*
>
> 793M /var/lib/db/couchdb
>
>
> .. running lsof shows the files marked as deleted but still there (I'm
> assuming beam still has some kind of handle still open to them) ..
>
>
> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>
> COMMAND PID USER FD TYPE DEVICE SIZE
> NODE NAME
>
> beam.smp 21352 couchdb 12u REG 253,10 4183
> 1210374
>
> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>
> beam.smp 21352 couchdb 16u REG 253,10 612039080
> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 20u REG 253,10 180699242
> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 21u REG 253,10 3133540
> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>
> beam.smp 21352 couchdb 25u REG 253,10 71422052
> 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 26u REG 253,10 71372900
> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>
> beam.smp 21352 couchdb 28u REG 253,10 621252717
> 1204238 /var/lib/db/couchdb/active_context.couch
>
> beam.smp 21352 couchdb 29u REG 253,10 1740903
> 90113 /var/lib/db/couchdb/mapping_cache.couch
>
> beam.smp 21352 couchdb 30u REG 253,10 3133540
> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>
> beam.smp 21352 couchdb 31u REG 253,10 2318439
> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>
> beam.smp 21352 couchdb 32u REG 253,10 983150
> 2004994
> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>
> beam.smp 21352 couchdb 33u REG 253,10 3133540
> 2056194 /var/lib/db/couchdb/thesaurus.couch
>
> beam.smp 21352 couchdb 34u REG 253,10 70742116
> 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 44u REG 253,10 1740903
> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>
> beam.smp 21352 couchdb 45u REG 253,10 65396836
> 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 47u REG 253,10 71229540
> 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 48u REG 253,10 240254078
> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 49u REG 253,10 2232423
> 1204241 /var/lib/db/couchdb/mappings.couch
>
> beam.smp 21352 couchdb 51u REG 253,10 96903353
> 1210371
>
> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>
> beam.smp 21352 couchdb 52u REG 253,10 1192034
> 1249282
>
> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>
> beam.smp 21352 couchdb 53u REG 253,10 1622135
> 2004993
> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>
> beam.smp 21352 couchdb 54u REG 253,10 4172
> 1208327
> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>
> beam.smp 21352 couchdb 55u REG 253,10 13496483
> 1208325
> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>
> beam.smp 21352 couchdb 56u REG 253,10 12959902
> 1208326
> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>
> beam.smp 21352 couchdb 57u REG 253,10 70254692
> 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 58u REG 253,10 581730
> 1210373
>
> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>
> beam.smp 21352 couchdb 67u REG 253,10 1818727
> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>
> beam.smp 21352 couchdb 78u REG 253,10 609905146
> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 80u REG 253,10 545919085
> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 81u REG 253,10 71032932
> 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>
> beam.smp 21352 couchdb 84u REG 253,10 548866111
> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 85u REG 253,10 564612402
> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 87u REG 253,10 615996179
> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>
> beam.smp 21352 couchdb 102u REG 253,10 561111149
> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>
>
> Has anyone seen this behaviour before or knows of a way around this ?
> Currently we have to restart the couchdb service in order to restore the
> disk space.
>
> Any help/advice is much appreciated.
>
> Regards
>
> Stephen
>
RE: Couch not releasing deleted files
Posted by Panop Suvaphrom <PS...@vizrt.com>.
Yeah, me too. On couch 1.0.1 with ubuntu 10.04
Do you have any idea to operate on this ?
-----Original Message-----
From: [mRg] [mailto:emargee@gmail.com]
Sent: Thursday, September 09, 2010 8:25 PM
To: user@couchdb.apache.org
Subject: Couch not releasing deleted files
[CouchDB Version : 0.11]
[OS: RHEL 5.5]
Hi all,
We are seeing some odd behaviour with CouchDb not releasing files after it
has compacted them meaning we are seeing disk space continue to increase
until the service is restarted even though we have regular compaction /
cleanup.
Typical scenario : File system alert for disk space ..
df says 5.5.GB used ..
df -h | grep -P 'Filesystem|db'
Filesystem Size Used Avail Use% Mounted on
8.0G 5.5G 2.2G 72% /var/lib/db
.. but the files on the disk (and in Futon) say only 793MB is being used ..
du -sh /var/lib/db/*
793M /var/lib/db/couchdb
.. running lsof shows the files marked as deleted but still there (I'm
assuming beam still has some kind of handle still open to them) ..
lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
COMMAND PID USER FD TYPE DEVICE SIZE
NODE NAME
beam.smp 21352 couchdb 12u REG 253,10 4183
1210374
/var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
beam.smp 21352 couchdb 16u REG 253,10 612039080
1204227 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 20u REG 253,10 180699242
1204237 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 21u REG 253,10 3133540
2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
beam.smp 21352 couchdb 25u REG 253,10 71422052
2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 26u REG 253,10 71372900
2056193 /var/lib/db/couchdb/active_context_schedule.couch
beam.smp 21352 couchdb 28u REG 253,10 621252717
1204238 /var/lib/db/couchdb/active_context.couch
beam.smp 21352 couchdb 29u REG 253,10 1740903
90113 /var/lib/db/couchdb/mapping_cache.couch
beam.smp 21352 couchdb 30u REG 253,10 3133540
1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
beam.smp 21352 couchdb 31u REG 253,10 2318439
1204240 /var/lib/db/couchdb/mappings.couch (deleted)
beam.smp 21352 couchdb 32u REG 253,10 983150
2004994
/var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
beam.smp 21352 couchdb 33u REG 253,10 3133540
2056194 /var/lib/db/couchdb/thesaurus.couch
beam.smp 21352 couchdb 34u REG 253,10 70742116
1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 44u REG 253,10 1740903
2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
beam.smp 21352 couchdb 45u REG 253,10 65396836
1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 47u REG 253,10 71229540
1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 48u REG 253,10 240254078
1204226 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 49u REG 253,10 2232423
1204241 /var/lib/db/couchdb/mappings.couch
beam.smp 21352 couchdb 51u REG 253,10 96903353
1210371
/var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
beam.smp 21352 couchdb 52u REG 253,10 1192034
1249282
/var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
beam.smp 21352 couchdb 53u REG 253,10 1622135
2004993
/var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
beam.smp 21352 couchdb 54u REG 253,10 4172
1208327
/var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
beam.smp 21352 couchdb 55u REG 253,10 13496483
1208325
/var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
beam.smp 21352 couchdb 56u REG 253,10 12959902
1208326
/var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
beam.smp 21352 couchdb 57u REG 253,10 70254692
1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 58u REG 253,10 581730
1210373
/var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
beam.smp 21352 couchdb 67u REG 253,10 1818727
1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
beam.smp 21352 couchdb 78u REG 253,10 609905146
1204234 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 80u REG 253,10 545919085
1994753 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 81u REG 253,10 71032932
1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
beam.smp 21352 couchdb 84u REG 253,10 548866111
1204239 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 85u REG 253,10 564612402
1204233 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 87u REG 253,10 615996179
1204228 /var/lib/db/couchdb/active_context.couch (deleted)
beam.smp 21352 couchdb 102u REG 253,10 561111149
2023425 /var/lib/db/couchdb/active_context.couch (deleted)
Has anyone seen this behaviour before or knows of a way around this ?
Currently we have to restart the couchdb service in order to restore the
disk space.
Any help/advice is much appreciated.
Regards
Stephen