You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2017/04/04 21:19:42 UTC
[jira] [Commented] (COUCHDB-3302) Attachment replication over low
bandwidth network connections
[ https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15955842#comment-15955842 ]
ASF subversion and git services commented on COUCHDB-3302:
----------------------------------------------------------
Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/COUCHDB-3288-remove-public-db-record from [~vatamane]
[ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ]
Prevent attachment upload from timing out during update_docs fabric call
Currently if an attachment was large enough or the connection was slow enough
such that it took more than fabric.request_timeout = 60000 milliseconds, the
fabric request would time out during attachment data transfer from coordinator
node to other nodes and the whole request would fail.
This was most evident when replicating database with large attachments.
The fix is to periodically send `attachment_chunk_received` to coordinator to
prevent the timeout.
COUCHDB-3302
> Attachment replication over low bandwidth network connections
> -------------------------------------------------------------
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit network connection (simulated locally with traffic shaping, see way below for an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator -------- Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0> -------- Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false" failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
> [{couch_att,'-foldl/4-fun-0-',3,
> [{file,"src/couch_att.erl"},{line,591}]},
> {couch_att,fold_streamed_data,4,
> [{file,"src/couch_att.erl"},{line,642}]},
> {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
> {couch_httpd_multipart,atts_to_mp,4,
> [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
> {gen_server,call,
> [<0.15778.0>,
> {send_req,
> {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false",
> "127.0.0.1",5983,undefined,undefined,
> "/t/doc1?new_edits=false",http,ipv4_address},
> [{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
> "multipart/related; boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
> put,
> {#Fun<couch_replicator_api_wrap.11.59841038>,
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
> [{att,<<"att_64">>,<<"application/octet-stream">>,
> 33193656,33193656,
> <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
> 161,242>>,
> 1,
> {follows,<0.8720.0>,#Ref<0.0.1.23804>},
> identity}],
> <<"0dea87076009b928b191e0b456375c93">>,33194202}},
> [{response_format,binary},
> {inactivity_timeout,30000},
> {socket_options,[{keepalive,true},{nodelay,false}]}],
> infinity}},
> infinity]}}}}
> {noformat}
> Expected Behaviour:
> Replication eventually succeeds.
> Appendix:
> Set up Traffic Shaping on a Mac:
> {noformat}
> (cat /etc/pf.conf && echo "dummynet-anchor \"reptest\"" && echo "anchor \"reptest\"") | sudo pfctl -f -
> echo "dummynet in quick proto tcp from any to any port 5983 pipe 3" | sudo pfctl -a reptest -f -
> echo "dummynet out quick proto tcp from any to any port 5983 pipe 4" | sudo pfctl -a reptest -f -
> sudo dnctl pipe 3 config bw 2Mbit/s
> sudo dnctl pipe 4 config bw 2Mbit/s
> sudo pfctl -E
> {noformat}
> Reset with:
> {noformat}
> sudo pfctl -f /etc/pf.conf
> sudo dnctl flush
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)