You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "sergey-safarov (via GitHub)" <gi...@apache.org> on 2026/03/21 16:55:07 UTC
[I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
sergey-safarov opened a new issue, #5934:
URL: https://github.com/apache/couchdb/issues/5934
### Version
unknown
### Describe the problem you're encountering
In my case CouchDB database used as configuration storage.
Here about 1000 configuration docs and some of them actively created, deleted and then pruned.
I found issue where all configuration docs required about 1Mb on filesystem but shard size on more then 500Mb.
To troubleshoot this I have deleted all documents in the database, compacted but still use more then 500 Mb on the filesystem.
Here is database info
```
[root@host bin]# curl -s http://admin:xxxx@127.0.0.1:5984/test| jq
{
"instance_start_time": "1768133824",
"db_name": "test",
"purge_seq": "4463449-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBEG1lq",
"update_seq": "8971555-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMHU-Vc4Gi7OYGRilGxpaEzcgCABfAIew",
"sizes": {
"file": 1322799314,
"external": 0,
"active": 508996585
},
"props": {},
"doc_del_count": 0,
"doc_count": 0,
"disk_format_version": 8,
"compact_running": true,
"cluster": {
"q": 1,
"n": 3,
"w": 2,
"r": 2
}
}
```
Please be aware here about `8 971 555` document changes and `4 463 449` purge docs.
To compact used command
```
[root@host bin]# curl -s -X POST -H "Content-Type: application/json" http://admin:xxxxx@127.0.0.1:5984/test/_compact| jq
{
"ok": true
}
```
On file system empty database use about 650 Mb
```
[root@esrp-0a bin]# ls -l /var/lib/couchdb/shards/00000000-ffffffff/test.1768133824.couch
-rw-r--r--. 1 couchdb couchdb 557266 Mar 21 16:38 /var/lib/couchdb/shards/00000000-ffffffff/test.1768133824.couch
[root@esrp-0a bin]# du -hs /var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design
157M /var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design
```
Shards info
```
[root@host bin]# curl -s http://admin:xxxxx@127.0.0.1:5984/test/_shards| jq
{
"shards": {
"00000000-ffffffff": [
"couchdb@node0.example.com",
"couchdb@node1.example.com",
"couchdb@node2.example.com"
]
}
}
```
Shard location
```
[root@node bin]# curl -s http://admin:xxxxx@127.0.0.1:5984/_node/_local/_dbs/test| jq
{
"_id": "test",
"_rev": "1-b562eb40b3a9250a1cee5d08d337a567",
"shard_suffix": [
46,
49,
55,
54,
56,
49,
51,
51,
56,
50,
52
],
"changelog": [
[
"add",
"00000000-ffffffff",
"couchdb@node0.example.com"
],
[
"add",
"00000000-ffffffff",
"couchdb@node1.example.com"
],
[
"add",
"00000000-ffffffff",
"couchdb@node2.example.com"
]
],
"by_node": {
"couchdb@node0.example.com": [
"00000000-ffffffff"
],
"couchdb@node1.example.com": [
"00000000-ffffffff"
],
"couchdb@node2.example.com": [
"00000000-ffffffff"
]
},
"by_range": {
"00000000-ffffffff": [
"couchdb@node0.example.com",
"couchdb@node1.example.com",
"couchdb@node2.example.com"
]
},
"props": {}
}
```
Cluster status
```
[root@host bin]# curl -s http://admin:xxxx@127.0.0.1:5984/_membership| jq
{
"all_nodes": [
"couchdb@node0.example.com",
"couchdb@node1.example.com",
"couchdb@node2.example.com"
],
"cluster_nodes": [
"couchdb@node0.example.com",
"couchdb@node1.example.com",
"couchdb@node2.example.com"
]
}
```
Could you suggest how to troubleshoot issue with user space cleanup when all docs deleted and pruned.
### Expected Behaviour
When all documents deleted from database shard files should be compacted and shard file size should be not more then 3Mb.
### Steps to Reproduce
It looks like issue related to cluster deployments and many documents created, deleted and then pruned.
### Your Environment
Three node cluster with CouchdB 3.5.0
```
[root@node bin]# curl -s http://admin:xxxx@127.0.0.1:5984| jq
{
"couchdb": "Welcome",
"version": "3.5.0",
"git_sha": "11f0d36",
"uuid": "3212a9b297b73cfe646f2c45dd7b8049",
"features": [
"access-ready",
"partitioned",
"pluggable-storage-engines",
"reshard",
"scheduler"
],
"vendor": {
"name": "The Apache Software Foundation"
}
}
```
### Additional Context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4273309318
yes we log compactions events.
Here is example
```
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.67402006.7> -------- Compaction started for db: shards/f8000000-ffffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622 idx: _design/maintenance
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9495.0> -------- ratio_views: Started compaction for shards/f8000000-ffffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622 _design/maintenance
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.60953491.7> -------- Compaction swap for view ./data/.shards/f8000000-ffffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622_design/mrview/2a0b045e46755c32dc9223d83c7ef1d5.view 1052980 114996
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.67402006.7> -------- Compaction finished for db: shards/f8000000-ffffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622 idx: _design/maintenance
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9495.0> -------- ratio_views: Starting compaction for shards/40000000-47ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622 _design/maintenance
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.65372365.7> -------- Compaction started for db: shards/40000000-47ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622 idx: _design/maintenance
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9495.0> -------- ratio_views: Started compaction for shards/40000000-47ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622 _design/maintenance
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.61586495.7> -------- Compaction swap for view ./data/.shards/40000000-47ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622_design/mrview/2a0b045e46755c32dc9223d83c7ef1d5.view 1052980 102708
Apr 17 09:59:30 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.65372365.7> -------- Compaction finished for db: shards/40000000-47ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622 idx: _design/maintenance
```
I also greped CouchDB logs for 2 days and do not see any strings containing "failed" word.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4356970551
> If you add a new document that should appear in the view to the db then query the view what happens? Any error or exceptions?
I have added document and fetched view and found just added document in the view output. But in view output still present non existent more documents.
No error in CouchDB logs.
I also made an other test.
In my use case view document named `_design/conferences`. I have copied view content into buffer and inserted to the new view document with name `_design/conferences2` (`_rev` key was deleted).
Then I pulled `_design/conferences2` view and still see not existed documents in the output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4241580087
> Are there any running [tasks](https://docs.couchdb.org/en/stable/api/server/common.html#active-tasks)? Does your log file show anything obvious?
Here is output
```sh
[root@esrp-0c ~]# curl 'http://admin:xxxxx@127.0.0.1:5984/_active_tasks'
[]
```
When starting compaction using command
```sh
[user@host bin]# curl -X POST \
> -H "Content-Type: application/json" \
> 'http://admin:xxxxx@127.0.0.1:5984/test/_compact'
{"ok":true}
```
In CouchDB logs I seelot of messages like
```
not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
```
As I understanding default compaction interval configured for 1 day (86400 seconds). And in current example compaction not running 92 days (7987469 seconds). And manual call does not triggers compaction. I have executed "compaction" command several timers.
Full couchdb logs follows.
```
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.236874151.6> -------- Starting compaction for db "shards/00000000-ffffffff/test.1768133824" at 12210749
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-17d6f419d28e73c696e0387c1a708051 not updated in 3201950 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-1a17e88c928130d58ac08b679a0d6d5b not updated in 380824 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-1d6c3e5997ef68eb1e3bfb850d37d24c not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-1e82cd03b50adbbc82a804b1cb64c3f2 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-20fef36bdec4b109a7cdc9ea2f3dfd70 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-24d66490155943d5ce5f8ba41c8f2cb6 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-27fcce55829ac4f09ab0f195d43ad529 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-2b65a5cdc3a494fa17cc1f645768def1 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-32266a2c7ee2f84ac7e384dce0086b62 not updated in 303408 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-34d7a15781b2915621611836b7e4ebd1 not updated in 2932127 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-462e71ffa4b5d3b38b811d08ad72396c not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-4fcb65e3071ca7c224d2ad2bfb0c4ba5 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-5ff8c2e39476fa5f8599fec47ae51e28 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-61afbb8b4289ba4cade0e41718010571 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-638c9523ae7c1e00fb22a48fa2eeb9d7 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-66d760f898ee98e42442c7d60470a7fa not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-6ca561964a095a15f145d0aac875a6da not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-78179a23e773efa34428d4ec9699d546 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-8e934f1da05d42acc4f18ae06e229bbc not updated in 380972 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-93cd52debd470f1a3463645e4d9225fc not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-94b03392130cc67dda8b6b56c27f9d7a not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-a413f1fd8747be194bd9a49a35aebcae not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-a6f995f290a08406e7b011d773a348b9 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-b50f2c6516787a57733d498dd67f3af7 not updated in 134479 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-b8d70e49452187e758d54b6387a7a718 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-b9a49bb8b24a083e07c436f17397fb54 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-c5dba9bbf7c3518033bc5d831e6f3eed not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-c994df8441b8f634949d2abcb0247cd6 not updated in 379251 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-d30620bd01fae48b31b30ecc821d76d3 not updated in 3202271 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-d338be563c1cba2f710f697553a19afb not updated in 381425 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-d46e5ec17c9392dd065d5cc4d21c6555 not updated in 381425 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-d7c66c68ec860660a8b01e99e2580ce4 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-dc04c69d4c0410ad3a7d7ab5cf7a1453 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-dccaf47c49f9992e74b0174177c4770a not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-dd51edd8663a1cc90af2f8215cd797be not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-e963fa424dfcd3a1903eb5d9c50ba41a not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-efec41fb4e3d3778431ffd14ed0a0ee6 not updated in 301785 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-f99e863b45527504b0efe0847be824c9 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-fd55255f67347619d4c1970e433773a5 not updated in 381425 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-fd96aadee2e2f27c57c72ac19945bf7f not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- The purge sequence for 'shards/00000000-ffffffff/test.1768133824' exceeds configured threshold
Apr 14 05:37:04 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248644112.6> fd1d212a34 undefined 2605:84c0:40:3::3b89 undefined GET /_up 200 ok 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4241752473
There's no compaction interval, compaction is triggered whenever needed, unless you've configured smoosh so it's not allowed to run.
So, we at least know that compaction is not finishing, which is why disk space is not being released. Additionally it seems you do a lot of purging and it's implied that you haven't been compacting for quite some time.
For today I suggest you just replicate this database to a new database and make sure the compaction daemon is configured to run (the defaults are fine). Compact that new database manually and you should not see the errors you just showed. Delete the old database once you've switched over to the new database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4356811420
@nickva you referenced #5709 two times.
Probable you want add reference to second PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4240016272
sorry, I see your 'after' status. the file size is the same as before (190021895). That pretty much has to mean the compaction task is not completely successfully. check your log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4427585222
> I have added document and fetched view and found document present in the view output. But in view output still present non existent more documents.
Does this happen on your system with a new view and without any purging. Just curious if purging has anything to do with this or not. Would it be possible to create a script to duplicate this behavior. "Create db, create view, update views..."
> @nickva you referenced https://github.com/apache/couchdb/pull/5709 two times.
Probable you want add reference to second PR.
Oh you're right, Sergey, my apologies, I meant to reference https://github.com/apache/couchdb/pull/5913
> Then I pulled _design/conferences2 view and still see not existed documents in the output.
In CouchDB we have a feature where the same view definition in the same db will generate the same view signature (it's a hash of the view/reduce functions) so renaming the design document will just point to the exact same view (It's a neat feature actually to allow updating indexes in production: create new view with ddoc_new id then, let it build in the background then replace ddoc with ddoc_new in production all the clients would see the already built view)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4536268493
@sergey-safarov thanks for giving it a try. We should have new rpms releases with the 3.5.2 version
```
│ ├── centos-10
│ │ ├── couchdb-3.5.2-1.el10.aarch64.rpm
│ │ └── couchdb-3.5.2-1.el10.x86_64.rpm
│ ├── centos-8
│ │ └── couchdb-3.5.2-1.el8.x86_64.rpm
│ ├── centos-9
│ │ ├── couchdb-3.5.2-1.el9.aarch64.rpm
│ │ └── couchdb-3.5.2-1.el9.x86_64.rpm
```
(from our jfrog repo ^)
The reason you were seeing 3.5.1 on main was that we hadn't yet merged the 3.5.2 updates from 3.5.x back into main. We just did that yesterday in https://github.com/apache/couchdb/pull/6014 so from now on it should show up as 3.5.2
In your test do you always start from a new db and go through the same steps - create views, fill the data, get the views, purge, compact (same steps) and wondering if you could share the view. Also wonder if you'd see the same view inconsistency without any purging taking place.
Ideally it would be great we had a reproducer script to replicate this behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4279115058
does your log include lines for the database compaction finishing too or only the views?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4280773198
Then send compaction command and get the output like[ here](https://github.com/apache/couchdb/issues/5934#issuecomment-4241580087)
What is important here present string
> The purge sequence for 'shards/00000000-ffffffff/test.1768133824' exceeds configured threshold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov closed issue #5934: [Bug]: used space do not freed after docs deleted and pruned
URL: https://github.com/apache/couchdb/issues/5934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4319942900
On other database I see similar issue.
When I calling `/_compact` endpoint in CouchDB logs I see
```
couchdb@host.example.org <0.126722554.0> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/mydb">> : _local/purge-mrview-e963fa424dfcd3a1903eb5d9c50ba41a not updated in 32383475 seconds. Expected update interval is at least 86400 seconds
```
And what is interest when pool view like
```
curl \
http://admin:xxxxx@127.0.0.1:5984/mydb/_design/conferences/_view/crossbar_listing
```
I see in response strings like this
```
{"id":"1-44sw5uqDFXmVPNewUsdBoyCrbkPedbyk","key":"1-44sw5uqDFXmVPNewUsdBoyCrbkPedbyk","value":{"id":"1-44sw5uqDFXmVPNewUsdBoyCrbkPedbyk","name":"Conference for call1056083289173202612242@192.168.169.11","member":{"pins":[],"numbers":[],"join_muted":false,"join_deaf":false},"moderator":{"pins":[],"numbers":[],"join_muted":false,"join_deaf":false},"conference_numbers":[],"flags":["rfc-4579"]}},
{"id":"1-44sw5uqDFXmVPNewUsdBoyFCEPyZJ5nV","key":"1-44sw5uqDFXmVPNewUsdBoyFCEPyZJ5nV","value":{"id":"1-44sw5uqDFXmVPNewUsdBoyFCEPyZJ5nV","name":"Conference for call1264317855163202612758@192.168.169.11","member":{"pins":[],"numbers":[],"join_muted":false,"join_deaf":false},"moderator":{"pins":[],"numbers":[],"join_muted":false,"join_deaf":false},"conference_numbers":[],"flags":["rfc-4579"]}}
```
But when I try get document present in the view response using command like
```
curl \
http://admin:xxxxx@127.0.0.1:5984/mydb/1-44sw5uqDFXmVPNewUsdBoyCrbkPedbyk
```
I got response
```
{"error":"not_found","reason":"missing"}
```
It looks like document already deleted but view does not track it about 374 days (32383475 seconds).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4336627014
we believe there are relevant fixes in couchdb main. We're working on a point release now. Once that's out, we'd like you to upgrade to it, invoke the view cleanup endpoint, and see if this resolves the matter.
cc @nickva
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4337340801
@sergey-safarov we do build packages for each `main` CI run: https://ci-couchdb.apache.org/job/jenkins-cm1/job/FullPlatformMatrix/job/main/1949/artifact/pkgs/
However I don't think we build RPMs aarch64 by default. We have bookworm arm64 ones but otherwise all RPMs (RHEL 10, 9, 8) and Debian bullseye/trixie, Ubuntu jammy/noble are all x86_64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4236485888
please manually invoke compaction for the database, confirm this was successfully completed, and report the sizes again, then we can investigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4279531988
Can you
1) query each of the views you have defined in this database to ensure they are up to date (one query per design document is enough, but do not do stale=ok or update=false)
2) call `POST /db/_view_cleanup`
and then try the database compaction again please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4244282376
Thanks @rnewson for suggetion.
Yes this works, after first replication database size about 1Mb as expected.
But on production server it required to switch client traffic between database current and replicated.
For this we use `HAproxy` daemon and some injected rules in `HAproxy` config file.
But this requires some skills and understanding.
This works for us and we use this as workaround for now.
This ticket I have created to troubleshot root of issue.
> There's no compaction interval, compaction is triggered whenever needed, unless you've configured smoosh so it's not allowed to run.
We use this `smoosh` settings
```
[smoosh]
state_dir = ./data
compaction_log_level = info
[smoosh.ratio_dbs]
priority = ratio
min_priority = 2.0
from = 08:00
to = 10:00
[smoosh.ratio_views]
priority = ratio
min_priority = 2.0
from = 08:00
to = 10:00
[smoosh.slack_dbs]
priority = slack
min_priority = 536870912
from = 08:00
to = 10:00
[smoosh.slack_views]
priority = slack
min_priority = 536870912
from = 08:00
to = 10:00
```
We use this settings to define day interval when we prefer compaction run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4244535258
did your log contain lines like `Compaction finished for db`? or `Compaction failed for db`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4344920644
I think it could be a combination of:
https://github.com/apache/couchdb/pull/5709
and maybe
https://github.com/apache/couchdb/pull/5709
In the first case we avoid creating checkpoints for invalid mrviews. This could mean views with no map functions or ddocs for non-mrview views (for example a clouseau doc previously would have also created an non-functional mrview purge checkpoint because we still managed to parse an #mrst{} record from it)
But those would only help for newly created dbs. If you already have invalid checkpoint it won't auto-clean them
It would also be interesting to see if those "stale" view design documents look like. To get purge the checkpoints you'd have to use something like `/test-db/_local_docs?include_system=true` but even then you may miss some as those are not clustered, maybe run that a few times or another way is to query 5986 (node local) shards/ individual shards. Checkpoints do not directly refer to ddocs I believe so we may need to run through all the design documents check their `/_info` checkpoint and cross reference them with the signature from.
> It looks like document already deleted but view does not track it about 374 days (32383475 seconds).
> {"error":"not_found","reason":"missing"} It looks like document already deleted but view does not track it about 374 days (32383475 seconds).
That I think is a smoking gun there is something off with the view building in general. If you add a new document that should appear in the view to the db then query the view what happens? Any error or exceptions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4336747994
We can update using RPM packages for aarch64.
Is it packages will be available for download?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4793033867
I have attempted to write test to reproduce issue on master, but no luck.
Let close this for now.
We have refactored our app to avoid temporal document creation, deletion and prune.
So we not able to reproduce this again.
Thanks for your help to troubleshoot this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4280957988
To make sure view was rebuild I have deleted folder
> /var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design/
And then send request to pull every view.
After this operation database size is same.
I also create `/opt/couchdb/etc/local.d/purge.ini` file with content
```
[couchdb]
purge_infos_limit = 50000000
```
And then triggered again 'compaction' and still see message
> The purge sequence for 'shards/00000000-ffffffff/test.1768133824' exceeds configured threshold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4279230451
I have filtered couchdb logs using command
```
journalctl -lu couchdb | grep ompact | grep -v view | tail -n 15
```
Command output will be
```
Apr 17 16:26:31 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/20000000-27ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 9.9375
Apr 17 16:26:32 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/70000000-77ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 10.75
Apr 17 16:26:42 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/a0000000-a7ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 11.0625
Apr 17 16:26:47 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/c8000000-cfffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 9.6875
Apr 17 16:26:55 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/28000000-2fffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 10.3125
Apr 17 16:27:01 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/b0000000-b7ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 10.25
Apr 17 16:27:08 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/b8000000-bfffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 10.375
Apr 17 16:27:09 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/48000000-4fffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 10.625
Apr 17 16:27:10 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/50000000-57ffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 10.625
Apr 17 16:27:12 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/e8000000-efffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 10.25
Apr 17 16:27:38 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/d8000000-dfffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622">> to compact with priority 10.25
Apr 17 16:55:16 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/60000000-7fffffff/system_config.1680271995">> to compact with priority 17
Apr 17 17:00:24 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/60000000-7fffffff/system_config.1680271995">> to compact with priority 20
Apr 17 17:05:29 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/60000000-7fffffff/system_config.1680271995">> to compact with priority 22
Apr 17 17:11:10 db-0c.example.com couchdb[2141995]: couchdb@db-0c.example.com <0.9314.0> -------- ratio_dbs: enqueueing <<"shards/60000000-7fffffff/system_config.1680271995">> to compact with priority 24
```
For db compaction work for other database, but not for 'test' database where a lot of pruned docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4279291353
thank you. so the database compaction does not complete, and this seems to be because of the large number of purged documents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4239932952
for the database `"compact_running": true,` means it's still going, you don't get any space back until it finishes. monitor `/_active_tasks` until all the compaction jobs have finished, check your log file to make sure they completed successfully, and then check your disk space usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4235659772
I just reproduced issue again.
For now I have do not deleted all docs from the database.
Current database status
```
[user@host bin]$ curl -s 'http://admin:xxxxx@127.0.0.1:5984/test' | jq
{
"instance_start_time": "1768133824",
"db_name": "test",
"purge_seq": "6101027-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBdGCNq",
"update_seq": "12211142-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMu4KP5QJF2c0NjFKMjC0Jm5EFACQiIi8",
"sizes": {
"file": 190083335,
"external": 657156,
"active": 188159324
},
"props": {},
"doc_del_count": 1584,
"doc_count": 844,
"disk_format_version": 8,
"compact_running": false,
"cluster": {
"q": 1,
"n": 3,
"w": 2,
"r": 2
}
}
```
On file system empty database use about 250 Mb
```
[user@host bin]$ ls -l /var/lib/couchdb/shards/00000000-ffffffff/test.1768133824.couch
-rw-r--r--. 1 couchdb couchdb 190083335 Apr 13 09:06 /var/lib/couchdb/shards/00000000-ffffffff/test.1768133824.couch
[user@host bin]$ du -hs /var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design
74M /var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design
```
If I dump call db content to text file it will take about 800 kb.
```
[user@host bin]$ curl -s 'http://admin:change_me@127.0.0.1:5984/test/_all_docs?include_docs=true&attachments=true' > /tmp/dump.json
[user@host bin]$ ls -l /tmp/dump.json
-rw-rw-r--. 1 user user 829276 Apr 13 10:19 /tmp/dump.json
```
Could you help troubleshoot why db size on disk more then 190 timer bigger then expected and cannot be compacted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4236488035
and manually compact all view groups too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "big-r81 (via GitHub)" <gi...@apache.org>.
big-r81 commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4238520099
Are there any running [tasks](https://docs.couchdb.org/en/stable/api/server/common.html#active-tasks)? Does your log file show anything obvious?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov closed issue #5934: [Bug]: used space do not freed after docs deleted and pruned
URL: https://github.com/apache/couchdb/issues/5934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4524686594
I have compiled rpm packages for for current master branch (commit a006a2ca161e4a82e6632247572f981c0de33037, for soem reason it reporting 3.5.1) and installed on servers.
```sh
[ssafarov@esrp-0a ~]$ curl -s http://admin:change_me@127.0.0.1:5984| jq
{
"couchdb": "Welcome",
"version": "3.5.1-a006a2c",
"git_sha": "a006a2c",
"uuid": "63d840e5b0823e4b275a71d25205d04c",
"features": [
"quickjs",
"access-ready",
"partitioned",
"pluggable-storage-engines",
"reshard",
"scheduler"
],
"vendor": {
"name": "The Apache Software Foundation"
}
}
```
During first seconds after start I see messages like
```
mem3_rep : deleting inactive purge checkpoint shards/38000000-3fffffff/account/e2/41/f5926a230d6bb1488de0f3c37cc7.1776064622 : _local/purge-mem3-23b49475b4082844d4b8ba06ab8964c0-ae267d1169b7e8dc5eaa983a9360008b
```
Then I send command compact views
```sh
curl -s -X POST -H "Content-Type: application/json" http://admin:xxxx@127.0.0.1:5984/test/_compact| jq
```
And get in the couchdb logs same message
```
couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-6ca561964a095a15f145d0aac875a6da not updated in 11364129 seconds. Expected update interval is at least 86400 seconds```
> create new view with ddoc_new id then, let it build in the background then replace ddoc with ddoc_new in production all the clients would see the already built view)
I have tested this method and it work in this case. This allow repair broken view.
And I execute compact
```sh
curl -s -X POST -H "Content-Type: application/json" http://admin:xxxx@127.0.0.1:5984/test/_compact| jq
```
And do not see more "deleting inactive purge checkpoint shard" messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4524693615
It looks like in last test.
Messages "deleting inactive purge checkpoint shard" still there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4103728752
Closing this.
Database freed while I creating ticket.
```
[root@host bin]# curl -s http://admin:xxxxx@127.0.0.1:5984/test| jq
{
"instance_start_time": "1768133824",
"db_name": "test",
"purge_seq": "4463449-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBEG1lq",
"update_seq": "8971555-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMHU-Vc4Gi7OYGRilGxpaEzcgCABfAIew",
"sizes": {
"file": 557266,
"external": 0,
"active": 138096
},
"props": {},
"doc_del_count": 0,
"doc_count": 0,
"disk_format_version": 8,
"compact_running": false,
"cluster": {
"q": 1,
"n": 3,
"w": 2,
"r": 2
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4244571813
note you'll need `info` level logging set under `[log]` section to get those
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4280676733
I have prepared script `ping_views.sh` with content
```sh
#!/bin/sh
URL="http://127.0.0.1:5984"
USERNAME="admin"
PASSWORD="xxxxx"
DBNAME="test"
jfile="/tmp/output.json"
#set -x
get_views_obj() {
curl -s -u ${USERNAME}:${PASSWORD} "${URL}/${DBNAME}/_all_docs?startkey=%22_design%2F%22&endkey=%22_design0%22&include_docs=true" | jq -c '.rows | map({"doc": .id, "views": (.doc.views | keys)})'
}
get_request_url_part() {
jq -r '.[] | .doc as $d | .views[] | "\($d)/_view/\(.)"' ${jfile}
}
# collect design document and view names as json object
get_views_obj > ${jfile}
# format design document and view names as url part
request_parts=$(get_request_url_part)
# send request to CouchDB server
for i in ${request_parts}; do
curl -s -u ${USERNAME}:${PASSWORD} "${URL}/${DBNAME}/${i}"
done
```
Using this script I have pulled response for every design document and view.
After script execution send
```sh
curl -X POST -s -u admin:xxxx http://127.0.0.1:5984/test/_view_cleanup
```
And then checked database status. Database have the same size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4238120415
> please manually invoke compaction for the database, confirm this was successfully completed, and report the sizes again, then we can investigate.
Here is results
```
[user@host bin]# curl -X POST \
> -H "Content-Type: application/json" \
> 'http://admin:xxxxx@127.0.0.1:5984/test/_compact'
{"ok":true}
[user@host bin]# curl -s 'http://admin:xxxxx@127.0.0.1:5984/test' | jq
{
"instance_start_time": "1768133824",
"db_name": "test",
"purge_seq": "6101027-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBdGCNq",
"update_seq": "12211142-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMu4KP5QJF2c0NjFKMjC0Jm5EFACQiIi8",
"sizes": {
"file": 190021895,
"external": 657156,
"active": 188157519
},
"props": {},
"doc_del_count": 1584,
"doc_count": 844,
"disk_format_version": 8,
"compact_running": true,
"cluster": {
"q": 1,
"n": 3,
"w": 2,
"r": 2
}
}
```
> and manually compact all view groups too
Here is procedure
```sh
# get design docs
views=$(curl -s --noproxy '*' 'http://admin:xxxxx@127.0.0.1:5984/test/_all_docs' | jq -r '.rows | map(.id) | map(select(startswith("_design"))) | .[]')
# compact views
set -x
for view in ${views}; do
view_stripped=$(echo "${view}" | sed -e 's|_design/||')
curl -X POST \
-H "Content-Type: application/json" \
"http://admin:xxxxx@127.0.0.1:5984/test/_compact/${view_stripped}"
done
```
After compaction database status
```
[user@host bin]# curl -s http://admin:xxxxx@127.0.0.1:5984/test/| jq
{
"instance_start_time": "1768133824",
"db_name": "test",
"purge_seq": "6101027-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBdGCNq",
"update_seq": "12211142-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMu4KP5QJF2c0NjFKMjC0Jm5EFACQiIi8",
"sizes": {
"file": 190021895,
"external": 657156,
"active": 188157519
},
"props": {},
"doc_del_count": 1584,
"doc_count": 844,
"disk_format_version": 8,
"compact_running": false,
"cluster": {
"q": 1,
"n": 3,
"w": 2,
"r": 2
}
}
```
For it looks like after compaction issue still here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: used space do not freed after docs deleted and pruned [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #5934:
URL: https://github.com/apache/couchdb/issues/5934#issuecomment-4338379643
Please let me know commit/branch which need to be tested and we try arrange this test on our environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org