You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Frank Walter <fr...@gmx.net> on 2019/05/01 18:22:47 UTC
Shards-directory still big after deleting big database in Fauxton
Hello
I have CouchDB v2.3.1 (on Ubuntu 14.04) and I use it only for creating
Wikipedia databases with mwscrape.
My shards folder was too big, over 50 GB big, so I deleted one big db
(frwiki) which had 32 GB in Fauxton. That db is gone now.
After this, I thought now my shards folder should be about 20 GB but it
is still 52 GB.
I don't find any documentation about that in the CouchDB Doc.
I restarted CouchDB (/etc/init.d/couchdb restart) but nothing changes.
How can I reduce the size of shards? How can I get rid of this ghost-db?
My next step would be, if I cannot solve this issue, to uninstall
CouchDB 2.x and reinstall 1.x, because I dont need that feature of
cluster server anyway. I see only inconvenience for my use.
Thanks
frank
Re: Shards-directory still big after deleting big database in Fauxton
Posted by Frank Röhm <fr...@gmx.net>.
Hello
Since I upgrade to 2.x I had only more work with it. The 1.5 is much easier to handle.
My 1.5 installation is not reachable through internet, so I guess there is not a problem with missing security patches.
Thanks
Frank
Am 12. Mai 2019 15:30:48 MESZ schrieb Adam Kocoloski <ko...@apache.org>:
>Hi Frank,
>
>Thanks for the followup. Definitely appreciate that the clustering feature adds complexity and is not appropriate for everyone. The only problem with running 1.x is that we are not providing any updates at all to that release series - even security patches.
>
>Was there something in particular that led you to downgrade?
>
>Cheers, Adam
>
>> On May 11, 2019, at 3:42 AM, Frank Röhm <fr...@gmx.net> wrote:
>>
>> In the end I indeed downgraded to couchdb v1.5 because I don’t use all this cluster feature and prefer to handle one file for each db ;)
>> So all is running again but with couch 1.5 on Ubuntu 14.04
>> And with Futon instead of Fauxton.
>>
>> Thanks.
>>
>> frank
>>
>>> Am 02.05.2019 um 15:21 schrieb Jan Lehnardt <ma...@jan.io>:
>>>
>>> Glad this worked out. Quick tip then, unless you run this on an 8-core (or more) machine, you might want to look into reducing your q for this database. q=2 or $num_cores is a good rule of thumb. You can use our couch-continuum tool to migrate an existing db: https://npmjs.com/couch-continuum
>>>
>>> Cheers
>>> Jan
>>> —
>>>
>>>> On 2. May 2019, at 14:17, Frank Röhm <fr...@gmx.net> wrote:
>>>>
>>>> OK, I found it.
>>>> In the 8 shards subdirectories (from 00000000-1fffffff to e0000000-ffffffff) there was still 8 frwiki directories (frwiki.1510609658.couch) with each 5 GB.
>>>> I deleted them with:
>>>>
>>>> find . -name frwiki.1510609658.couch -delete
>>>>
>>>> from the shards dir and gone they are.
>>>> Hopefully it won’t affect my CouchDB, but as I heard this is very robust ;)
>>>>
>>>> I think I can stick to the v2.x now, no need to downgrade now, ouff.
>>>>
>>>> frank
>>>>
>>>>> Am 02.05.2019 um 07:25 schrieb Joan Touzet <wo...@apache.org>:
>>>>>
>>>>> Look for a .deleted directory under your data/ directory. The files may not have been deleted but moved aside due to the enable_database_recovery setting, or because the DB was still in use when you restarted CouchDB.
>>>>>
>>>>> Another useful command is:
>>>>>
>>>>> $ du -sh /opt/couchdb/data/*
>>>>>
>>>>> which should tell you where the storage is being used. Does this show anything useful to you?
>>>>>
>>>>> -Joan
>>>>>
>>>>>> On 2019-05-01 2:22 p.m., Frank Walter wrote:
>>>>>> Hello
>>>>>> I have CouchDB v2.3.1 (on Ubuntu 14.04) and I use it only for creating
>>>>>> Wikipedia databases with mwscrape.
>>>>>> My shards folder was too big, over 50 GB big, so I deleted one big db
>>>>>> (frwiki) which had 32 GB in Fauxton. That db is gone now.
>>>>>> After this, I thought now my shards folder should be about 20 GB but it
>>>>>> is still 52 GB.
>>>>>> I don't find any documentation about that in the CouchDB Doc.
>>>>>> I restarted CouchDB (/etc/init.d/couchdb restart) but nothing changes.
>>>>>> How can I reduce the size of shards? How can I get rid of this ghost-db?
>>>>>> My next step would be, if I cannot solve this issue, to uninstall
>>>>>> CouchDB 2.x and reinstall 1.x, because I dont need that feature of
>>>>>> cluster server anyway. I see only inconvenience for my use.
>>>>>> Thanks
>>>>>> frank
>>>>
>>>
>>
>
Re: Shards-directory still big after deleting big database in Fauxton
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Frank,
Thanks for the followup. Definitely appreciate that the clustering feature adds complexity and is not appropriate for everyone. The only problem with running 1.x is that we are not providing any updates at all to that release series - even security patches.
Was there something in particular that led you to downgrade?
Cheers, Adam
> On May 11, 2019, at 3:42 AM, Frank Röhm <fr...@gmx.net> wrote:
>
> In the end I indeed downgraded to couchdb v1.5 because I don’t use all this cluster feature and prefer to handle one file for each db ;)
> So all is running again but with couch 1.5 on Ubuntu 14.04
> And with Futon instead of Fauxton.
>
> Thanks.
>
> frank
>
>> Am 02.05.2019 um 15:21 schrieb Jan Lehnardt <ma...@jan.io>:
>>
>> Glad this worked out. Quick tip then, unless you run this on an 8-core (or more) machine, you might want to look into reducing your q for this database. q=2 or $num_cores is a good rule of thumb. You can use our couch-continuum tool to migrate an existing db: https://npmjs.com/couch-continuum
>>
>> Cheers
>> Jan
>> —
>>
>>> On 2. May 2019, at 14:17, Frank Röhm <fr...@gmx.net> wrote:
>>>
>>> OK, I found it.
>>> In the 8 shards subdirectories (from 00000000-1fffffff to e0000000-ffffffff) there was still 8 frwiki directories (frwiki.1510609658.couch) with each 5 GB.
>>> I deleted them with:
>>>
>>> find . -name frwiki.1510609658.couch -delete
>>>
>>> from the shards dir and gone they are.
>>> Hopefully it won’t affect my CouchDB, but as I heard this is very robust ;)
>>>
>>> I think I can stick to the v2.x now, no need to downgrade now, ouff.
>>>
>>> frank
>>>
>>>> Am 02.05.2019 um 07:25 schrieb Joan Touzet <wo...@apache.org>:
>>>>
>>>> Look for a .deleted directory under your data/ directory. The files may not have been deleted but moved aside due to the enable_database_recovery setting, or because the DB was still in use when you restarted CouchDB.
>>>>
>>>> Another useful command is:
>>>>
>>>> $ du -sh /opt/couchdb/data/*
>>>>
>>>> which should tell you where the storage is being used. Does this show anything useful to you?
>>>>
>>>> -Joan
>>>>
>>>>> On 2019-05-01 2:22 p.m., Frank Walter wrote:
>>>>> Hello
>>>>> I have CouchDB v2.3.1 (on Ubuntu 14.04) and I use it only for creating
>>>>> Wikipedia databases with mwscrape.
>>>>> My shards folder was too big, over 50 GB big, so I deleted one big db
>>>>> (frwiki) which had 32 GB in Fauxton. That db is gone now.
>>>>> After this, I thought now my shards folder should be about 20 GB but it
>>>>> is still 52 GB.
>>>>> I don't find any documentation about that in the CouchDB Doc.
>>>>> I restarted CouchDB (/etc/init.d/couchdb restart) but nothing changes.
>>>>> How can I reduce the size of shards? How can I get rid of this ghost-db?
>>>>> My next step would be, if I cannot solve this issue, to uninstall
>>>>> CouchDB 2.x and reinstall 1.x, because I dont need that feature of
>>>>> cluster server anyway. I see only inconvenience for my use.
>>>>> Thanks
>>>>> frank
>>>
>>
>
Re: Shards-directory still big after deleting big database in Fauxton
Posted by Frank Röhm <fr...@gmx.net>.
In the end I indeed downgraded to couchdb v1.5 because I don’t use all this cluster feature and prefer to handle one file for each db ;)
So all is running again but with couch 1.5 on Ubuntu 14.04
And with Futon instead of Fauxton.
Thanks.
frank
> Am 02.05.2019 um 15:21 schrieb Jan Lehnardt <ma...@jan.io>:
>
> Glad this worked out. Quick tip then, unless you run this on an 8-core (or more) machine, you might want to look into reducing your q for this database. q=2 or $num_cores is a good rule of thumb. You can use our couch-continuum tool to migrate an existing db: https://npmjs.com/couch-continuum
>
> Cheers
> Jan
> —
>
>> On 2. May 2019, at 14:17, Frank Röhm <fr...@gmx.net> wrote:
>>
>> OK, I found it.
>> In the 8 shards subdirectories (from 00000000-1fffffff to e0000000-ffffffff) there was still 8 frwiki directories (frwiki.1510609658.couch) with each 5 GB.
>> I deleted them with:
>>
>> find . -name frwiki.1510609658.couch -delete
>>
>> from the shards dir and gone they are.
>> Hopefully it won’t affect my CouchDB, but as I heard this is very robust ;)
>>
>> I think I can stick to the v2.x now, no need to downgrade now, ouff.
>>
>> frank
>>
>>> Am 02.05.2019 um 07:25 schrieb Joan Touzet <wo...@apache.org>:
>>>
>>> Look for a .deleted directory under your data/ directory. The files may not have been deleted but moved aside due to the enable_database_recovery setting, or because the DB was still in use when you restarted CouchDB.
>>>
>>> Another useful command is:
>>>
>>> $ du -sh /opt/couchdb/data/*
>>>
>>> which should tell you where the storage is being used. Does this show anything useful to you?
>>>
>>> -Joan
>>>
>>>> On 2019-05-01 2:22 p.m., Frank Walter wrote:
>>>> Hello
>>>> I have CouchDB v2.3.1 (on Ubuntu 14.04) and I use it only for creating
>>>> Wikipedia databases with mwscrape.
>>>> My shards folder was too big, over 50 GB big, so I deleted one big db
>>>> (frwiki) which had 32 GB in Fauxton. That db is gone now.
>>>> After this, I thought now my shards folder should be about 20 GB but it
>>>> is still 52 GB.
>>>> I don't find any documentation about that in the CouchDB Doc.
>>>> I restarted CouchDB (/etc/init.d/couchdb restart) but nothing changes.
>>>> How can I reduce the size of shards? How can I get rid of this ghost-db?
>>>> My next step would be, if I cannot solve this issue, to uninstall
>>>> CouchDB 2.x and reinstall 1.x, because I dont need that feature of
>>>> cluster server anyway. I see only inconvenience for my use.
>>>> Thanks
>>>> frank
>>
>
Re: Shards-directory still big after deleting big database in Fauxton
Posted by Jan Lehnardt <ma...@jan.io>.
Glad this worked out. Quick tip then, unless you run this on an 8-core (or more) machine, you might want to look into reducing your q for this database. q=2 or $num_cores is a good rule of thumb. You can use our couch-continuum tool to migrate an existing db: https://npmjs.com/couch-continuum
Cheers
Jan
—
> On 2. May 2019, at 14:17, Frank Röhm <fr...@gmx.net> wrote:
>
> OK, I found it.
> In the 8 shards subdirectories (from 00000000-1fffffff to e0000000-ffffffff) there was still 8 frwiki directories (frwiki.1510609658.couch) with each 5 GB.
> I deleted them with:
>
> find . -name frwiki.1510609658.couch -delete
>
> from the shards dir and gone they are.
> Hopefully it won’t affect my CouchDB, but as I heard this is very robust ;)
>
> I think I can stick to the v2.x now, no need to downgrade now, ouff.
>
> frank
>
>> Am 02.05.2019 um 07:25 schrieb Joan Touzet <wo...@apache.org>:
>>
>> Look for a .deleted directory under your data/ directory. The files may not have been deleted but moved aside due to the enable_database_recovery setting, or because the DB was still in use when you restarted CouchDB.
>>
>> Another useful command is:
>>
>> $ du -sh /opt/couchdb/data/*
>>
>> which should tell you where the storage is being used. Does this show anything useful to you?
>>
>> -Joan
>>
>>> On 2019-05-01 2:22 p.m., Frank Walter wrote:
>>> Hello
>>> I have CouchDB v2.3.1 (on Ubuntu 14.04) and I use it only for creating
>>> Wikipedia databases with mwscrape.
>>> My shards folder was too big, over 50 GB big, so I deleted one big db
>>> (frwiki) which had 32 GB in Fauxton. That db is gone now.
>>> After this, I thought now my shards folder should be about 20 GB but it
>>> is still 52 GB.
>>> I don't find any documentation about that in the CouchDB Doc.
>>> I restarted CouchDB (/etc/init.d/couchdb restart) but nothing changes.
>>> How can I reduce the size of shards? How can I get rid of this ghost-db?
>>> My next step would be, if I cannot solve this issue, to uninstall
>>> CouchDB 2.x and reinstall 1.x, because I dont need that feature of
>>> cluster server anyway. I see only inconvenience for my use.
>>> Thanks
>>> frank
>
Re: Shards-directory still big after deleting big database in Fauxton
Posted by Frank Röhm <fr...@gmx.net>.
OK, I found it.
In the 8 shards subdirectories (from 00000000-1fffffff to e0000000-ffffffff) there was still 8 frwiki directories (frwiki.1510609658.couch) with each 5 GB.
I deleted them with:
find . -name frwiki.1510609658.couch -delete
from the shards dir and gone they are.
Hopefully it won’t affect my CouchDB, but as I heard this is very robust ;)
I think I can stick to the v2.x now, no need to downgrade now, ouff.
frank
> Am 02.05.2019 um 07:25 schrieb Joan Touzet <wo...@apache.org>:
>
> Look for a .deleted directory under your data/ directory. The files may not have been deleted but moved aside due to the enable_database_recovery setting, or because the DB was still in use when you restarted CouchDB.
>
> Another useful command is:
>
> $ du -sh /opt/couchdb/data/*
>
> which should tell you where the storage is being used. Does this show anything useful to you?
>
> -Joan
>
> On 2019-05-01 2:22 p.m., Frank Walter wrote:
>> Hello
>> I have CouchDB v2.3.1 (on Ubuntu 14.04) and I use it only for creating
>> Wikipedia databases with mwscrape.
>> My shards folder was too big, over 50 GB big, so I deleted one big db
>> (frwiki) which had 32 GB in Fauxton. That db is gone now.
>> After this, I thought now my shards folder should be about 20 GB but it
>> is still 52 GB.
>> I don't find any documentation about that in the CouchDB Doc.
>> I restarted CouchDB (/etc/init.d/couchdb restart) but nothing changes.
>> How can I reduce the size of shards? How can I get rid of this ghost-db?
>> My next step would be, if I cannot solve this issue, to uninstall
>> CouchDB 2.x and reinstall 1.x, because I dont need that feature of
>> cluster server anyway. I see only inconvenience for my use.
>> Thanks
>> frank
Re: Shards-directory still big after deleting big database in Fauxton
Posted by Joan Touzet <wo...@apache.org>.
Look for a .deleted directory under your data/ directory. The files may
not have been deleted but moved aside due to the
enable_database_recovery setting, or because the DB was still in use
when you restarted CouchDB.
Another useful command is:
$ du -sh /opt/couchdb/data/*
which should tell you where the storage is being used. Does this show
anything useful to you?
-Joan
On 2019-05-01 2:22 p.m., Frank Walter wrote:
> Hello
>
> I have CouchDB v2.3.1 (on Ubuntu 14.04) and I use it only for creating
> Wikipedia databases with mwscrape.
> My shards folder was too big, over 50 GB big, so I deleted one big db
> (frwiki) which had 32 GB in Fauxton. That db is gone now.
> After this, I thought now my shards folder should be about 20 GB but it
> is still 52 GB.
>
> I don't find any documentation about that in the CouchDB Doc.
> I restarted CouchDB (/etc/init.d/couchdb restart) but nothing changes.
>
> How can I reduce the size of shards? How can I get rid of this ghost-db?
>
> My next step would be, if I cannot solve this issue, to uninstall
> CouchDB 2.x and reinstall 1.x, because I dont need that feature of
> cluster server anyway. I see only inconvenience for my use.
>
> Thanks
>
> frank
>