You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Ramkrishna Kulkarni <ra...@gmail.com> on 2019/05/07 04:28:50 UTC
Replication fails for DB with large number of zero size attachments
(1.7.1 to 2.3.1)
Hello,
I have a DB with tens of thousands of documents and each document has
around 10 to 15 zero size attachments. Earlier these attachments were
around 1 MB each but I had to move them to S3 but still had to give the
impression these attachments existed in the DB. So I uploaded empty
attachments.
Now I want to migrate to 2.3.1 from 1.7.1 but this particular DB fails to
complete replication. All other DBs which do not have attachments got
replicated within a minute. Sometimes it gets stuck after around 2 to 5
thousand documents and some times it throws errors like the one pasted
below.
--- error ---
[error] 2019-05-07T04:12:44.396103Z nonode@nohost <0.4602.0> 36e8dc35e5
req_err(965732698) unknown_error : normal
[<<"mochiweb_request:recv/3
L223">>,<<"mochiweb_request:stream_unchunked_body/5
L581">>,<<"mochiweb_request:recv_body/2 L257">>,<<"chttpd:body/1
L655">>,<<"chttpd:json_body/1 L668">>,<<"chttpd:json_body_obj/1
L676">>,<<"chttpd_db:db_req/2 L399">>,<<"chttpd:handle_req_after_auth/2
L320">>]
[notice] 2019-05-07T04:12:44.396991Z nonode@nohost <0.4602.0> 36e8dc35e5
172.21.0.3:5984 172.21.0.2 ram POST /batuadm_apps/_bulk_docs 500 ok 5
[notice] 2019-05-07T04:12:55.622900Z nonode@nohost <0.4985.0> d678ae28f3
172.21.0.3:5984 172.21.0.2 ram PUT
/batuadm_apps/03da47e5f1d027cdf13dde7c4a0a8562?new_edits=false 201 ok 300013
[error] 2019-05-07T04:12:55.629143Z nonode@nohost <0.4985.0> f4c4127550
req_err(965732698) unknown_error : normal
[<<"mochiweb_request:recv/3
L223">>,<<"mochiweb_request:stream_unchunked_body/5
L581">>,<<"mochiweb_request:recv_body/2 L257">>,<<"chttpd:body/1
L655">>,<<"chttpd:json_body/1 L668">>,<<"chttpd:json_body_obj/1
L676">>,<<"chttpd_db:db_req/2 L399">>,<<"chttpd:handle_req_after_auth/2
L320">>]
[notice] 2019-05-07T04:12:55.630624Z nonode@nohost <0.4985.0> f4c4127550
172.21.0.3:5984 172.21.0.2 ram POST /batuadm_apps/_bulk_docs 500 ok 6
[notice] 2019-05-07T04:12:55.642848Z nonode@nohost <0.4986.0> 64d1dc7f35
172.21.0.3:5984 172.21.0.2 ram PUT
/batuadm_apps/03da47e5f1d027cdf13dde7c4a09b80c?new_edits=false 201 ok 300004
[error] 2019-05-07T04:12:55.649938Z nonode@nohost <0.4986.0> 50d2654213
req_err(965732698) unknown_error : normal
[<<"mochiweb_request:recv/3
L223">>,<<"mochiweb_request:stream_unchunked_body/5
L581">>,<<"mochiweb_request:recv_body/2 L257">>,<<"chttpd:body/1
L655">>,<<"chttpd:json_body/1 L668">>,<<"chttpd:json_body_obj/1
L676">>,<<"chttpd_db:db_req/2 L399">>,<<"chttpd:handle_req_after_auth/2
L320">>]
[notice] 2019-05-07T04:12:55.650862Z nonode@nohost <0.4986.0> 50d2654213
172.21.0.3:5984 172.21.0.2 ram POST /batuadm_apps/_bulk_docs 500 ok 7
--- error --
Intermittently I also see some documents getting "PUT" but the replication
never seems to complete even after 10-15 hours. Both CouchDB instances are
on the same server (Docker containers).
--- put ---
[notice] 2019-05-07T04:17:43.200867Z nonode@nohost <0.4415.0> c73ab3d049
172.21.0.3:5984 172.21.0.2 ram PUT
/batuadm_apps/03da47e5f1d027cdf13dde7c4a01d214?new_edits=false 201 ok 300003
[notice] 2019-05-07T04:17:43.246857Z nonode@nohost <0.4416.0> bcd5f7f9f8
172.21.0.3:5984 172.21.0.2 ram PUT
/batuadm_apps/03da47e5f1d027cdf13dde7c4a0a4519?new_edits=false 201 ok 300003
...
--- put ---
Any help would be much appreciated.
Thanks.
Re: Replication fails for DB with large number of zero size
attachments (1.7.1 to 2.3.1)
Posted by Ramkrishna Kulkarni <ra...@gmail.com>.
Hello,
This is how I got it to finish finally. I replicated 100 documents at a
time by explicitly specifying the document ids. After a couple of rounds or
those, I did direct DB replicate and it succeeded.
Thanks.
On Tue, May 7, 2019 at 9:58 AM Ramkrishna Kulkarni <
ramkrishna.kulkarni@gmail.com> wrote:
> Hello,
>
> I have a DB with tens of thousands of documents and each document has
> around 10 to 15 zero size attachments. Earlier these attachments were
> around 1 MB each but I had to move them to S3 but still had to give the
> impression these attachments existed in the DB. So I uploaded empty
> attachments.
>
> Now I want to migrate to 2.3.1 from 1.7.1 but this particular DB fails to
> complete replication. All other DBs which do not have attachments got
> replicated within a minute. Sometimes it gets stuck after around 2 to 5
> thousand documents and some times it throws errors like the one pasted
> below.
>
>
> --- error ---
> [error] 2019-05-07T04:12:44.396103Z nonode@nohost <0.4602.0> 36e8dc35e5
> req_err(965732698) unknown_error : normal
> [<<"mochiweb_request:recv/3
> L223">>,<<"mochiweb_request:stream_unchunked_body/5
> L581">>,<<"mochiweb_request:recv_body/2 L257">>,<<"chttpd:body/1
> L655">>,<<"chttpd:json_body/1 L668">>,<<"chttpd:json_body_obj/1
> L676">>,<<"chttpd_db:db_req/2 L399">>,<<"chttpd:handle_req_after_auth/2
> L320">>]
> [notice] 2019-05-07T04:12:44.396991Z nonode@nohost <0.4602.0> 36e8dc35e5
> 172.21.0.3:5984 172.21.0.2 ram POST /batuadm_apps/_bulk_docs 500 ok 5
> [notice] 2019-05-07T04:12:55.622900Z nonode@nohost <0.4985.0> d678ae28f3
> 172.21.0.3:5984 172.21.0.2 ram PUT
> /batuadm_apps/03da47e5f1d027cdf13dde7c4a0a8562?new_edits=false 201 ok 300013
> [error] 2019-05-07T04:12:55.629143Z nonode@nohost <0.4985.0> f4c4127550
> req_err(965732698) unknown_error : normal
> [<<"mochiweb_request:recv/3
> L223">>,<<"mochiweb_request:stream_unchunked_body/5
> L581">>,<<"mochiweb_request:recv_body/2 L257">>,<<"chttpd:body/1
> L655">>,<<"chttpd:json_body/1 L668">>,<<"chttpd:json_body_obj/1
> L676">>,<<"chttpd_db:db_req/2 L399">>,<<"chttpd:handle_req_after_auth/2
> L320">>]
> [notice] 2019-05-07T04:12:55.630624Z nonode@nohost <0.4985.0> f4c4127550
> 172.21.0.3:5984 172.21.0.2 ram POST /batuadm_apps/_bulk_docs 500 ok 6
> [notice] 2019-05-07T04:12:55.642848Z nonode@nohost <0.4986.0> 64d1dc7f35
> 172.21.0.3:5984 172.21.0.2 ram PUT
> /batuadm_apps/03da47e5f1d027cdf13dde7c4a09b80c?new_edits=false 201 ok 300004
> [error] 2019-05-07T04:12:55.649938Z nonode@nohost <0.4986.0> 50d2654213
> req_err(965732698) unknown_error : normal
> [<<"mochiweb_request:recv/3
> L223">>,<<"mochiweb_request:stream_unchunked_body/5
> L581">>,<<"mochiweb_request:recv_body/2 L257">>,<<"chttpd:body/1
> L655">>,<<"chttpd:json_body/1 L668">>,<<"chttpd:json_body_obj/1
> L676">>,<<"chttpd_db:db_req/2 L399">>,<<"chttpd:handle_req_after_auth/2
> L320">>]
> [notice] 2019-05-07T04:12:55.650862Z nonode@nohost <0.4986.0> 50d2654213
> 172.21.0.3:5984 172.21.0.2 ram POST /batuadm_apps/_bulk_docs 500 ok 7
> --- error --
>
>
> Intermittently I also see some documents getting "PUT" but the replication
> never seems to complete even after 10-15 hours. Both CouchDB instances are
> on the same server (Docker containers).
>
> --- put ---
> [notice] 2019-05-07T04:17:43.200867Z nonode@nohost <0.4415.0> c73ab3d049
> 172.21.0.3:5984 172.21.0.2 ram PUT
> /batuadm_apps/03da47e5f1d027cdf13dde7c4a01d214?new_edits=false 201 ok 300003
> [notice] 2019-05-07T04:17:43.246857Z nonode@nohost <0.4416.0> bcd5f7f9f8
> 172.21.0.3:5984 172.21.0.2 ram PUT
> /batuadm_apps/03da47e5f1d027cdf13dde7c4a0a4519?new_edits=false 201 ok 300003
> ...
> --- put ---
>
> Any help would be much appreciated.
>
> Thanks.
>
>