You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by eiri <gi...@git.apache.org> on 2016/02/23 17:18:54 UTC
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
GitHub user eiri opened a pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26
Fix view filtered replication
This fixes the filtered by view replication for the views built without seq_indexed option.
I kept `couch_replicator_api_wrap:get_view_info/3` function for a back reference even though we can't use it right now in the clustered environment as it's not marshalled through fabric.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloudant/couchdb-couch-replicator 57286-fix-view-based-replication
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/couchdb-couch-replicator/pull/26.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #26
----
commit dac3e09c860336c4f2268c385da865040cf7e86a
Author: Eric Avdey <ei...@eiri.ca>
Date: 2016-02-23T14:13:06Z
Add tests for a view filtred replication
commit 60f201b7193fc82975f9dbc1991d015a17daffdb
Author: Eric Avdey <ei...@eiri.ca>
Date: 2016-02-23T14:15:47Z
Fix get_view_info function
commit d27aa267b020f41dd64c88b32bfff5998c213842
Author: Eric Avdey <ei...@eiri.ca>
Date: 2016-02-23T15:51:07Z
Don't use update_seq from view info for filtered replication
The view's update_seq is not applicable to the database changes feed
and in general meaninless in the clustered environment
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-188365001
IIRC @rnewson warned that couch shouldn't call any fabric bits since it's should be in reverse. May be we can just rework @chewbranca patch in other way to fix that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-187793339
Hm...what's the difference in 17 and 18 erlangs that could cause that? o.O
I'll try to fallback to my plan B to find the point when this problem happened. Because there it wasn't exists till now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/couchdb-couch-replicator/pull/26
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by eiri <gi...@git.apache.org>.
Github user eiri commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-188482611
@chewbranca I agree that using ddoc cache is a good thing, not so much on calling fabric from couch, looks like an opportunity to fall into cycling calls. I'd prefer either have ddoc cache to support none clustered databases or change the replicator (and its tests) to be cluster aware.
Anyway, I've got the tests passing locally with inserting the test's ddoc into ddoc_cache in `couch_replicator_filtered_tests` and adding the following clause in `couch_changes.erl`
```
filter(Db, DocInfo, {fetch, Style, Req, {DesignId, _}, FName}) ->
{ok, DDoc} = ddoc_cache:open_doc(fabric:dbname(Db), DesignId),
filter(Db, DocInfo, {custom, Style, Req, DDoc, FName}).
```
This looks rather hacky, so if you have a better solution I'm all ears.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-187809630
@eiri ok, I can say for sure it all started since https://github.com/apache/couchdb-couch-replicator/commit/b18ff40d3adde03f4f85f9acff4950ddbb3b2e06
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-187810521
But fun fact: #24 build is green. However, by rolling back commit by commit that problem is gone after b18ff40. Any ideas?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by eiri <gi...@git.apache.org>.
Github user eiri commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-189300365
Squashed, retested, merged
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-187816863
@eiri Don't blame you at all! (:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by eiri <gi...@git.apache.org>.
Github user eiri commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-188337027
@kxepal : well, I found it. the failure is happening because of [couchdb-couch/fabc25f](https://github.com/apache/couchdb-couch/commit/fabc25f351e2a30d2f7d8ba2131595707b156e22?diff-8df5f7bc5f26a88ee4ec43baeadfb84cR233#diff-8df5f7bc5f26a88ee4ec43baeadfb84cR233)
The call to `ddoc_cache:open_doc/2` in there ends up calling `fabric:open_doc/3` which is trying to load shards from `shards_db` and throwing `database_does_not_exist`, because we are not running the replication tests on a sharded database.
I need to think how to work around this, though it is unsettling that we are casually leaking fabric into couchdb-couch.
CC: @chewbranca
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by janl <gi...@git.apache.org>.
Github user janl commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-190817399
> Squashed, retested, merged
Boom! :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by eiri <gi...@git.apache.org>.
Github user eiri commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-187792054
@kxepal This is weird: the test are running fine for me on Debian with R17, but failing on MacOS with R18.2. This is happening with prior of this PR, i.e. on the master, but I'm pretty sure I've ran the tests successfully a couple of weeks ago. The real error for the reader to die is that it seems can't find source db, I see `database_does_not_exist` thrown from `fabric_doc_open`.
I'll dig into it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-187792356
Some error logging (change tmp/etc/eunit.ini log level):
```
module 'couch_replicator_filtered_tests'
Filtered replication tests
[error] ChangesReader process died with reason: {database_does_not_exist,
[{mem3_shards,
load_shards_from_db,
"eunit-test-db-1456246859461347",
[{file,
"src/mem3_shards.erl"},
{line,282}]},
{mem3_shards,
load_shards_from_disk,1,
[{file,
"src/mem3_shards.erl"},
{line,270}]},
{mem3_shards,
load_shards_from_disk,2,
[{file,
"src/mem3_shards.erl"},
{line,286}]},
{mem3_shards,for_docid,3,
[{file,
"src/mem3_shards.erl"},
{line,85}]},
{fabric_doc_open,go,3,
[{file,
"src/fabric_doc_open.erl"},
{line,38}]},
{couch_changes,
configure_filter,4,
[{file,
"src/couch_changes.erl"},
{line,235}]},
{couch_changes,
handle_changes,4,
[{file,
"src/couch_changes.erl"},
{line,68}]},
{couch_replicator_api_wrap,
changes_since,5,
[{file,
"src/couch_replicator_api_wrap.erl"},
{line,529}]}]}
[error] Replication `1af0a180102905d0583caa669b582cc6` (`eunit-test-db-1456246859461347` -> `eunit-test-db-145624686042079`) failed: changes_reader_died
undefined
*** instantiation of subtests failed ***
**in function couch_replicator_filtered_tests:should_succeed/2 (test/couch_replicator_filtered_tests.erl, line 103)
**error:{badmatch,{error,changes_reader_died}}
[error] ChangesReader process died with reason: {database_does_not_exist,
[{mem3_shards,
load_shards_from_db,
"eunit-test-db-1456246860226785",
[{file,
"src/mem3_shards.erl"},
{line,282}]},
{mem3_shards,
load_shards_from_disk,1,
[{file,
"src/mem3_shards.erl"},
{line,270}]},
{mem3_shards,
load_shards_from_disk,2,
[{file,
"src/mem3_shards.erl"},
{line,286}]},
{mem3_shards,for_docid,3,
[{file,
"src/mem3_shards.erl"},
{line,85}]},
{fabric_doc_open,go,3,
[{file,
"src/fabric_doc_open.erl"},
{line,38}]},
{couch_changes,
configure_filter,4,
[{file,
"src/couch_changes.erl"},
{line,235}]},
{couch_changes,
handle_changes,4,
[{file,
"src/couch_changes.erl"},
{line,68}]},
{couch_replicator_api_wrap,
changes_since,5,
[{file,
"src/couch_replicator_api_wrap.erl"},
{line,529}]}]}
[error] Replication `bf497b49e250885ec27023416af0bff9` (`eunit-test-db-1456246860226785` -> `http://127.0.0.1:38481/eunit-test-db-1456246860498725/`) failed: changes_reader_died
undefined
*** instantiation of subtests failed ***
**in function couch_replicator_filtered_tests:should_succeed/2 (test/couch_replicator_filtered_tests.erl, line 103)
**error:{badmatch,{error,changes_reader_died}}
[error] Uncaught error in HTTP request: {error,database_does_not_exist}
[error] httpd 500 error response:
{"error":"unknown_error","reason":"database_does_not_exist"}
[error] Replicator, request GET to "http://127.0.0.1:38481/eunit-test-db-1456246860747962/_changes?filter=filter_ddoc%2Ftestfilter&feed=normal&style=all_docs&since=0&timeout=10000" failed. The received HTTP error code is 500
[error] Uncaught error in HTTP request: {error,database_does_not_exist}
[error] httpd 500 error response:
{"error":"unknown_error","reason":"database_does_not_exist"}
[error] Replicator, request GET to "http://127.0.0.1:38481/eunit-test-db-1456246860747962/_changes?filter=filter_ddoc%2Ftestfilter&feed=normal&style=all_docs&since=0&timeout=10000" failed. The received HTTP error code is 500
[error] Uncaught error in HTTP request: {error,database_does_not_exist}
[error] httpd 500 error response:
{"error":"unknown_error","reason":"database_does_not_exist"}
[error] Replicator, request GET to "http://127.0.0.1:38481/eunit-test-db-1456246860747962/_changes?filter=filter_ddoc%2Ftestfilter&feed=normal&style=all_docs&since=0&timeout=10000" failed. The received HTTP error code is 500
[error] Uncaught error in HTTP request: {error,database_does_not_exist}
[error] httpd 500 error response:
{"error":"unknown_error","reason":"database_does_not_exist"}
[error] Replicator, request GET to "http://127.0.0.1:38481/eunit-test-db-1456246860747962/_changes?filter=filter_ddoc%2Ftestfilter&feed=normal&style=all_docs&since=0&timeout=10000" failed. The received HTTP error code is 500
[error] Uncaught error in HTTP request: {error,database_does_not_exist}
[error] httpd 500 error response:
{"error":"unknown_error","reason":"database_does_not_exist"}
[error] Replicator, request GET to "http://127.0.0.1:38481/eunit-test-db-1456246860747962/_changes?filter=filter_ddoc%2Ftestfilter&feed=normal&style=all_docs&since=0&timeout=10000" failed. The received HTTP error code is 500
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by chewbranca <gi...@git.apache.org>.
Github user chewbranca commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-188423675
@kxepal @eiri @rnewson IMO loading design docs from `ddoc_cache` is obviously the right thing to do. The big question here is whether to support non clustered databases in `ddoc_cache`.
One alternative approach to calling `ddoc_cache` directly in this scenario is to use `couch_changes:open_ddoc/2` [1] which would work around the issue we're seeing here. I think having to qualify loading from `ddoc_cache` is a hacky solution we should strive to avoid, and which is why I didn't do it initially, but it might be the pragmatic choice here until we eliminate this current dichotomy of APIs.
[1] https://github.com/apache/couchdb-couch/blob/master/src/couch_changes.erl#L341-L355
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-189264983
Hooray! +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by chewbranca <gi...@git.apache.org>.
Github user chewbranca commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-188949666
@eiri @kxepal @rnewson: came up with a work around: https://github.com/apache/couchdb-couch/commit/de509a6b1e3774e5cc57b49c2cffb9304cfb03c0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by eiri <gi...@git.apache.org>.
Github user eiri commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-187814596
@kxepal it started at b18ff40 because the tests were broken prior of that commit, they've just been not working, reporting false positive :smile:
I think it's not in replicator itself, but somewhere else. Right now I'm eyeing [couchdb-fabric/8bb97fc](https://github.com/apache/couchdb-fabric/commit/8bb97fcf59d44374e4e0516a33293d22a9c2308c), but I'm occupied at the moment, will get my hands on it in 40-50 minutes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-189124902
@chewbranca changes landed (thank you!). @eiri try to `git commit --amend && git push -f` to trigger commit hash change and let travis rebuild.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-187787335
Hm...still the same issue:
```
module 'couch_replicator_filtered_tests'
Filtered replication tests
undefined
*** instantiation of subtests failed ***
**in function couch_replicator_filtered_tests:should_succeed/2 (test/couch_replicator_filtered_tests.erl, line 103)
**error:{badmatch,{error,changes_reader_died}}
```
Did I miss something?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Fix view filtered replicati...
Posted by eiri <gi...@git.apache.org>.
Github user eiri commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/26#issuecomment-189260169
@kxepal It's green :green_book: And thanks to the discussion we had here now I know that the next step would be to add the same ddoc's caching for this type of a filter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---