You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "Jan Lehnardt (JIRA)" <ji...@apache.org> on 2017/02/16 17:49:42 UTC
[jira] [Created] (COUCHDB-3302) Attachment replication over low
bandwidth network connections
Jan Lehnardt created COUCHDB-3302:
-------------------------------------
Summary: Attachment replication over low bandwidth network connections
Key: COUCHDB-3302
URL: https://issues.apache.org/jira/browse/COUCHDB-3302
Project: CouchDB
Issue Type: Bug
Components: Replication
Reporter: Jan Lehnardt
Setup:
Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit network connection (simulated locally with traffic shaping, see way below for an example).
git clone https://github.com/apache/couchdb.git
cd couchdb
./configure --disable-docs --disable-fauxton
make release
cd ..
cp -r couchdb/rel/couchdb source
cp -r couchdb/rel/couchdb target
# set up local ini: chttpd / port: 5981 / 5983
# set up vm.args: source@hostname.local / target@hostname.local
# no admins
Start both CouchDB in their own terminal windows: ./bin/couchdb
# create all required databases, and our `t` test database
curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
# create 64MB attachments
dd if=/dev/urandom of=att-64 bs=1024 count=65536
# create doc on source
curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: application/octet-stream' -d @att-64
# replicate to target
curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
With the traffic shaping in place, the replication call doesn’t return, and eventually CouchDB fails with:
[error] 2017-02-16T17:37:30.488990Z couchdbtak@rose.local emulator -------- Error in process <0.15811.0> on node 'couchdbtak@rose.local' with exit value:
{{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
[error] 2017-02-16T17:37:30.490610Z couchdbtak@rose.local <0.8721.0> -------- Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false" failed due to error {error,
{'EXIT',
{{{nocatch,{mp_parser_died,noproc}},
[{couch_att,'-foldl/4-fun-0-',3,
[{file,"src/couch_att.erl"},{line,591}]},
{couch_att,fold_streamed_data,4,
[{file,"src/couch_att.erl"},{line,642}]},
{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
{couch_httpd_multipart,atts_to_mp,4,
[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
{gen_server,call,
[<0.15778.0>,
{send_req,
{{url,"http://127.0.0.1:5983/t/doc1?new_edits=false",
"127.0.0.1",5983,undefined,undefined,
"/t/doc1?new_edits=false",http,ipv4_address},
[{"Accept","application/json"},
{"Content-Length",33194202},
{"Content-Type",
"multipart/related; boundary=\"0dea87076009b928b191e0b456375c93\""},
{"User-Agent","CouchDB-Replicator/2.0.0"}],
put,
{#Fun<couch_replicator_api_wrap.11.59841038>,
{<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
[{att,<<"att_64">>,<<"application/octet-stream">>,
33193656,33193656,
<<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
161,242>>,
1,
{follows,<0.8720.0>,#Ref<0.0.1.23804>},
identity}],
<<"0dea87076009b928b191e0b456375c93">>,33194202}},
[{response_format,binary},
{inactivity_timeout,30000},
{socket_options,[{keepalive,true},{nodelay,false}]}],
infinity}},
infinity]}}}}
Expected:
Replication eventually succeeds.
Appendix:
Set up Traffic Shaping on a Mac:
(cat /etc/pf.conf && echo "dummynet-anchor \"reptest\"" && echo "anchor \"reptest\"") | sudo pfctl -f -
echo "dummynet in quick proto tcp from any to any port 5983 pipe 3" | sudo pfctl -a reptest -f -
echo "dummynet out quick proto tcp from any to any port 5983 pipe 4" | sudo pfctl -a reptest -f -
sudo dnctl pipe 3 config bw 2Mbit/s
sudo dnctl pipe 4 config bw 2Mbit/s
sudo pfctl -E
Reset with:
sudo pfctl -f /etc/pf.conf
sudo dnctl flush
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)