[GitHub] H--o-l edited a comment on issue #1621: Black matter in shard files

[GitBox <gi...@apache.org>]

H--o-l edited a comment on issue #1621: Black matter in shard files
URL: https://github.com/apache/couchdb/issues/1621#issuecomment-426681587
 
 
   Hi !
   
   I'm working with @adrienverge and I continue to investigate the "black matter" issue.
   
   As @adrienverge said, shard files (`db.1520354435.couch`) constantly keep growing (by several kB per day), even though there are no requests done to them.
   This is a problem because we have *many* small, untouched databases.
   
   Compaction helps to free space on a regular basis, but it takes a bunch of cpu time.
   So we would like to understand what is happening in our `.couch` files.
   
   In a node of our cluster we monitored output of `strings /var/lib/couchd/shards/xxx/db.1520354435.couch`.
   In one night, it add ~800 new rows like this one:
   ```
   kv_nodel
   ?_local/shard-sync-VNaRvSQnMLpUSzV8iZvcow-IOHL5NIdglwNQCT0DL_V0Ah
   seqa}	],
   target_uuid
    cef8fe0a2eac1f0f7af0d5e0c56fbbd8	7
   historyh
   @;couchdb@node1l
   142-216
   -$!7
   source
   x2a65d174d772861e4cd085e9e347729-
   $	timestamp!7|
   2018-10-02T22:21:13.181557Zjjjj	~Jd
   TgKo_QqiB9XPEFr08HvUC2Q
   98-214
   M/Uf|e0dc03c296d3847efd92e964053e5f1b	7
   P15:23:01.533200Zjjjjj
   kp_nodel
   ?_local/shard-sync-IOHL5NIdglwNQCT0DL_V0A-gKo_QqiB9XPEFr08HvUC2Qh
   -IOHLFj
   	db_headera
   	size_infob
   nilb
    cef8fe0a2eac1f0f7af0d5e0c56fbbd8l
   <couchdb@node2a
   ```
   
   As we have ~34k dbs, multiplied by 8 shards by node, it quickly sum up and takes a lot of space and a lot of compaction time, which sometimes ends up saturating our disks and CPU.
   
   Can you advise us ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services