[GitHub] [couchdb] rnewson edited a comment on issue #1470: Fix calculation of external size for attachments

[GitBox <gi...@apache.org>]

rnewson edited a comment on issue #1470: Fix calculation of external size for attachments
URL: https://github.com/apache/couchdb/pull/1470#issuecomment-469391866
 
 
   I'm struggling to reproduce that result, though we should note that compaction is an essential maintenance task and it clearly solves the problem.
   
   My attempt to isolate this was as follows;
   
   1) I created a single shard database;
   curl 'foo:bar@localhost:15984/db1?n=1&q=1' -XPUT
   
   2) "sizes" is
   
   curl 'foo:bar@localhost:15984/db1' -s | jq .sizes
   {
     "file": 8348,
     "external": 0,
     "active": 0
   }
   
   which looks like two couch file headers and an empty b+tree.
   
   2) I made a file exactly 10MB long
   dd if=/dev/zero of=zeroes bs=1m count=10
   
   3) sizes after inserting this with
   curl 'foo:bar@localhost:15984/db1/doc1/att1' -XPUT -T zeroes
   
   {
     "file": 10522824,
     "external": 10485764,
     "active": 10505529
   }
   
   4) I then updated the document, preserving the attachment
   
   {
     "file": 10547400,
     "external": 10485791,
     "active": 10513692
   }
   
   This confirms we do _not_ write an additional copy of the attachment, we're simply pointing at the original chunks from rev 1.
   
   5) I compact the db and sizes are;
   
   {
     "file": 10526923,
     "external": 10485786,
     "active": 10505598
   }
   
   ---
   
   Conclusion: There's no unexpected size bloat at any stage. 
   
   Over time, as databases open/close and when couchdb is restarted, couch will add a new 4k 'header' at the end of the file, but you'd need a lot of those to reach 30mb. Other changes to the database would lead to old versions of b+tree nodes being littered in the file, superseded by later writes.
   
   Can you do this test and tell us if you get a different result? Or perhaps there are other things going on in your situation that explain it.
   
   Again, compaction is a required maintenance step and it fixes the problem. We are concerned if there's excessive (and unexplained) write magnification so please let us know if you can reproduce this in isolation.
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services