You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2023/01/12 04:55:34 UTC
[GitHub] [couchdb] nickva opened a new pull request, #4370: Ensure design docs are upload individually when replicating with bulk_get
nickva opened a new pull request, #4370:
URL: https://github.com/apache/couchdb/pull/4370
Previously, when replication jobs used _bulk_get, they didn't upload design docs individually like they do when not using _bulk_get.
Add tests to cover both attachments and ddoc cases. meck:num_calls/3 is helpful as it allows to nicely assert which API function was called and how many times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a diff in pull request #4370: Ensure design docs are uploaded individually when replicating with bulk_get
Posted by GitBox <gi...@apache.org>.
nickva commented on code in PR #4370:
URL: https://github.com/apache/couchdb/pull/4370#discussion_r1068494445
##########
src/couch_replicator/src/couch_replicator_worker.erl:
##########
@@ -349,8 +349,9 @@ bulk_get(#httpdb{} = Source, #{} = IdRevs) ->
case couch_replicator_api_wrap:bulk_get(Source, IdRevs, Opts) of
{ok, #{} = Docs} ->
FilterFun = fun
- (_, #doc{atts = []}) -> true;
+ (_, #doc{id = <<?DESIGN_DOC_PREFIX, _/binary>>}) -> false;
Review Comment:
Updated as described in the note
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] rnewson commented on pull request #4370: Ensure design docs are uploaded individually when replicating with bulk_get
Posted by GitBox <gi...@apache.org>.
rnewson commented on PR #4370:
URL: https://github.com/apache/couchdb/pull/4370#issuecomment-1382003048
@nickva thanks for the links. It still seems to be a (perhaps necessary) hack around a problem elsewhere. We should do The Right Thing for any _bulk_doc write that writes multiple design documents. It is not right imo to just teach our replicator to avoid triggering those bugs, no other client will be similarly protected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #4370: Ensure design docs are uploaded individually when replicating with bulk_get
Posted by GitBox <gi...@apache.org>.
nickva commented on PR #4370:
URL: https://github.com/apache/couchdb/pull/4370#issuecomment-1380862162
Updated with the minor optimization and added references in the commit message and PR comment to the previous PR which introduced this behavior and the related issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] rnewson commented on pull request #4370: Ensure design docs are uploaded individually when replicating with bulk_get
Posted by GitBox <gi...@apache.org>.
rnewson commented on PR #4370:
URL: https://github.com/apache/couchdb/pull/4370#issuecomment-1379989587
why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #4370: Ensure design docs are uploaded individually when replicating with bulk_get
Posted by GitBox <gi...@apache.org>.
nickva commented on PR #4370:
URL: https://github.com/apache/couchdb/pull/4370#issuecomment-1382047815
@rnewson with the replicator we are trying to be compatible even with old, or possibly broken clients, clients. But most of all, in this PR, we are trying to bring back the previous behavior before the recent _bulk_get optimization. Currently, it's odd that the there is a difference based on whether `[replicator] use_bulk_get` is toggled or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a diff in pull request #4370: Ensure design docs are uploaded individually when replicating with bulk_get
Posted by GitBox <gi...@apache.org>.
nickva commented on code in PR #4370:
URL: https://github.com/apache/couchdb/pull/4370#discussion_r1068417710
##########
src/couch_replicator/src/couch_replicator_worker.erl:
##########
@@ -349,8 +349,9 @@ bulk_get(#httpdb{} = Source, #{} = IdRevs) ->
case couch_replicator_api_wrap:bulk_get(Source, IdRevs, Opts) of
{ok, #{} = Docs} ->
FilterFun = fun
- (_, #doc{atts = []}) -> true;
+ (_, #doc{id = <<?DESIGN_DOC_PREFIX, _/binary>>}) -> false;
Review Comment:
I see minor optimization here - since we know right off the bat these are design documents, we don't have to fetch them in bulk_get at all (unlike the ones with attachments). So am thinking of updating to fetch and upload design docs, fetch the bulk docs, do the attachments ones. What do we think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #4370: Ensure design docs are uploaded individually when replicating with bulk_get
Posted by GitBox <gi...@apache.org>.
nickva commented on PR #4370:
URL: https://github.com/apache/couchdb/pull/4370#issuecomment-1380718306
@rnewson good question, I should have explain that a bit better in the commit.
That is to preserve an already existing behavior which we had in the replicator without _bulk_get usage for more than 2 years. It was introduced here: https://github.com/apache/couchdb/pull/2426 related to these issues https://github.com/apache/couchdb/issues/2415 and https://github.com/apache/couchdb/issues/2413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva merged pull request #4370: Ensure design docs are uploaded individually when replicating with bulk_get
Posted by GitBox <gi...@apache.org>.
nickva merged PR #4370:
URL: https://github.com/apache/couchdb/pull/4370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org