You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2021/05/10 15:00:24 UTC
[GitHub] [couchdb] bessbd opened a new pull request #3553: Add error propagation to couch_eval and couch_views_indexer
bessbd opened a new pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553
<!-- Thank you for your contribution!
Please file this form by replacing the Markdown comments
with your text. If a section needs no action - remove it.
Also remember, that CouchDB uses the Review-Then-Commit (RTC) model
of code collaboration. Positive feedback is represented +1 from committers
and negative is a -1. The -1 also means veto, and needs to be addressed
to proceed. Once there are no objections, the PR can be merged by a
CouchDB committer.
See: http://couchdb.apache.org/bylaws.html#decisions for more info. -->
## Overview
When `ApiMod:acquire_map_context` returns with `{error, any()}`, the caller(s) fail with `badmatch`. This PR changes the caller hierarchy so that those `{error, _}` are propagated back and a failing `update` gets retried (up to `retry_limit` times).
## Testing recommendations
<!-- Describe how we can test your changes.
Does it provides any behaviour that the end users
could notice? -->
I tested manually: `curl -XPUT -HContent-type:application/json -d'{"views": { "testView": { "map": "function(doc) {if(doc.name) {emit(doc.name)};}" } }, "language": "javascript"}' http://adm:pass@127.0.0.1:15984/foo/_design/fooDD2` and `curl http://adm:pass@127.0.0.1:15984/foo/_design/fooDD2/_view/testView` and I watched as the `badmatch` log entries disappeared after the change.
## Related Issues or Pull Requests
<!-- If your changes affects multiple components in different
repositories please put links to those issues or pull requests here. -->
## Checklist
- [ ] Code is written and works correctly
- [ ] Changes are covered by tests
- [ ] Any new configurable parameters are documented in `rel/overlay/etc/default.ini`
- [ ] A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632443043
##########
File path: src/couch_views/test/couch_views_indexer_test.erl
##########
@@ -610,6 +613,23 @@ index_can_recover_from_crash(Db) ->
], Out).
+handle_acquire_map_context_error(_) ->
+ meck:new(mock_language_server, [non_strict]),
+ config:set("couch_eval.languages", ?QUERY_SERVER_LANG_STRING,
+ atom_to_list(mock_language_server)),
+ meck:expect(mock_language_server, acquire_map_context,
Review comment:
Done in 71f2765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
nickva commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632075860
##########
File path: src/couch_views/test/couch_views_indexer_test.erl
##########
@@ -610,6 +613,23 @@ index_can_recover_from_crash(Db) ->
], Out).
+handle_acquire_map_context_error(_) ->
+ meck:new(mock_language_server, [non_strict]),
+ config:set("couch_eval.languages", ?QUERY_SERVER_LANG_STRING,
+ atom_to_list(mock_language_server)),
+ meck:expect(mock_language_server, acquire_map_context,
Review comment:
Minor nit: usually the formatting for fun would look like:
```
meck:expect(mock_language_server, acquire_map_context, fun(_) ->
{error, foo_error}
end)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#issuecomment-836996618
@bessbd the error comes from trying to acquire a map context. Would this usually be considered a user error (bad javascript, values are too large, etc) or server-side error, as in we something is broken with the backend (operator misconfigured the language server backend). If the error is from a misconfigured query language server, wonder if failing the job after 3 tries would make sense. If the backend is fixed, would the user still get a failed error, if so it might be better to perhaps keep retrying longer.
How retries happens is a bit subtle too it seems... Notice when we retry we simply abandon the job. We exit the but still keep ownership of the job https://github.com/apache/couchdb/pull/3553/files#diff-228d2a0ae65ae417bf4b3fb19f0de9f886c2ae168ab9fbec8e5f86b93934e6f7R148. It would eventually timeout based on https://github.com/apache/couchdb/blob/c75a9b66965939e470ed5f3f2749626d3f35d311/src/couch_views/src/couch_views_jobs.erl#L33 and rely on the job activity monitor re-enqueuing the jobs. That's not the worst and it would provide a backoff of sorts but it certainly subtle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
nickva commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632077942
##########
File path: src/couch_eval/test/error_tests.erl
##########
@@ -0,0 +1,53 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(error_tests).
Review comment:
We usually want to use the application prefix in the module names, even for test modules. So it would be `couch_eval_error_tests`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
nickva commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632079503
##########
File path: src/couch_eval/test/error_tests.erl
##########
@@ -0,0 +1,53 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(error_tests).
+
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("couch/include/couch_db.hrl").
+
+-define(LANG_BINARY, <<"foo_lang">>).
+-define(LANG_STRING, binary_to_list(?LANG_BINARY)).
+
+
+setup() ->
+ meck:new(mock_language_server, [non_strict]),
+ Ctx = test_util:start_couch(),
+ config:set("couch_eval.languages", ?LANG_STRING,
+ atom_to_list(mock_language_server)),
+ Ctx.
+
+
+teardown(Ctx) ->
+ test_util:stop_couch(Ctx),
+ meck:unload().
+
+
+error_test_() ->
+ {
+ "Error tests",
+ {
+ setup,
+ fun setup/0, fun teardown/1,
+ [
+ fun acquire_map_context_error_handled/0
+ ]
+ }
+ }.
+
+acquire_map_context_error_handled() ->
+ meck:expect(mock_language_server, acquire_map_context,
+ fun(_) -> {error, foo_error} end),
+ Result = couch_eval:acquire_map_context(<<"foo">>, <<"bar">>, ?LANG_BINARY,
+ <<"baz">>, <<"quux">>, [<<"quuz">>]),
+ ?assertEqual({error, foo_error}, Result).
+
Review comment:
Minor formatting nit, let's use the standard 4 space indentation. [Emilio](https://github.com/cloudant-labs/emilio) can help us here
```
~/src/emilio/emilio src/couch_eval/test/error_tests.erl
src/couch_eval/test/error_tests.erl:23(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:24(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:25(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:27(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:31(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:32(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:36(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:39(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:40(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:41(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:43(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:45(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:45(5) - 302 - there should be 2 empty lines between functions, not 1
src/couch_eval/test/error_tests.erl:48(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:49(5) - 141 - inline fun does not start on call line
src/couch_eval/test/error_tests.erl:50(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:52(1) - 111 - leading white space should be a multiple of 4
src/couch_eval/test/error_tests.erl:53(1) - 210 - file ends with 2 new lines, not 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r631963116
##########
File path: src/couch_views/src/couch_views_indexer.erl
##########
@@ -561,15 +561,19 @@ start_query_server(#mrst{qserver = nil} = Mrst) ->
lib = Lib,
views = Views
} = Mrst,
- {ok, QServer} = couch_eval:acquire_map_context(
+ case couch_eval:acquire_map_context(
DbName,
DDocId,
Language,
Sig,
Lib,
[View#mrview.def || View <- Views]
- ),
- Mrst#mrst{qserver = QServer};
+ ) of
+ {ok, QServer} ->
+ Mrst#mrst{qserver = QServer};
+ {error, Error} ->
+ throw(Error)
Review comment:
Done in https://github.com/cloudant/nodejs-cloudant#initialization-callback . Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632446815
##########
File path: src/couch_eval/test/error_tests.erl
##########
@@ -0,0 +1,53 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(error_tests).
+
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("couch/include/couch_db.hrl").
+
+-define(LANG_BINARY, <<"foo_lang">>).
+-define(LANG_STRING, binary_to_list(?LANG_BINARY)).
+
+
+setup() ->
+ meck:new(mock_language_server, [non_strict]),
+ Ctx = test_util:start_couch(),
+ config:set("couch_eval.languages", ?LANG_STRING,
+ atom_to_list(mock_language_server)),
+ Ctx.
+
+
+teardown(Ctx) ->
+ test_util:stop_couch(Ctx),
+ meck:unload().
+
+
+error_test_() ->
+ {
+ "Error tests",
+ {
+ setup,
+ fun setup/0, fun teardown/1,
+ [
+ fun acquire_map_context_error_handled/0
+ ]
+ }
+ }.
+
+acquire_map_context_error_handled() ->
+ meck:expect(mock_language_server, acquire_map_context,
+ fun(_) -> {error, foo_error} end),
+ Result = couch_eval:acquire_map_context(<<"foo">>, <<"bar">>, ?LANG_BINARY,
+ <<"baz">>, <<"quux">>, [<<"quuz">>]),
+ ?assertEqual({error, foo_error}, Result).
+
Review comment:
Done in 890c1ea . Do you think we could integrate emilio to main CI to improve the review flow on the project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd merged pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd merged pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632442974
##########
File path: src/couch_views/test/couch_views_indexer_test.erl
##########
@@ -610,6 +613,23 @@ index_can_recover_from_crash(Db) ->
], Out).
+handle_acquire_map_context_error(_) ->
+ meck:new(mock_language_server, [non_strict]),
+ config:set("couch_eval.languages", ?QUERY_SERVER_LANG_STRING,
+ atom_to_list(mock_language_server)),
+ meck:expect(mock_language_server, acquire_map_context,
+ fun(_) -> {error, foo_error} end),
+ {'EXIT', {Reason, _}} = (catch couch_views_indexer:start_query_server(#mrst{
Review comment:
Done in 71f2765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd commented on pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#issuecomment-841611406
Thank you for the approve, @nickva ! I'm about to merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#issuecomment-841322154
@bessbd we did try to integrate it, there is a `make emilio` [check](https://github.com/apache/couchdb/blob/main/Makefile#L209). One tricky part is having to figure out warning for existing code vs new code. One day we could perhaps one large reformat pass and have everything compliant with emilio. There is a formatter we could use too: https://github.com/WhatsApp/erlfmt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632443236
##########
File path: src/couch_views/test/couch_views_indexer_test.erl
##########
@@ -610,6 +613,23 @@ index_can_recover_from_crash(Db) ->
], Out).
+handle_acquire_map_context_error(_) ->
+ meck:new(mock_language_server, [non_strict]),
+ config:set("couch_eval.languages", ?QUERY_SERVER_LANG_STRING,
+ atom_to_list(mock_language_server)),
+ meck:expect(mock_language_server, acquire_map_context,
+ fun(_) -> {error, foo_error} end),
+ {'EXIT', {Reason, _}} = (catch couch_views_indexer:start_query_server(#mrst{
Review comment:
Done in 71f2765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd commented on pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#issuecomment-841297969
> Looks good. Good idea to write tests. Just a few minor formatting issue then +1
Thank you for the review, @nickva . I believe I've pushed commits for all the changes you've requested. Also, I ran emilio on the problematic files so they should be good now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
bessbd commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632443752
##########
File path: src/couch_eval/test/error_tests.erl
##########
@@ -0,0 +1,53 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(error_tests).
Review comment:
Done in b7fb2cd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
nickva commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632074634
##########
File path: src/couch_views/test/couch_views_indexer_test.erl
##########
@@ -610,6 +613,23 @@ index_can_recover_from_crash(Db) ->
], Out).
+handle_acquire_map_context_error(_) ->
+ meck:new(mock_language_server, [non_strict]),
+ config:set("couch_eval.languages", ?QUERY_SERVER_LANG_STRING,
+ atom_to_list(mock_language_server)),
+ meck:expect(mock_language_server, acquire_map_context,
+ fun(_) -> {error, foo_error} end),
+ {'EXIT', {Reason, _}} = (catch couch_views_indexer:start_query_server(#mrst{
Review comment:
I think `?assertError(foo_error, ` might work better here.
Also as is, I think the line is longer than 80 columns. Could pull `Mrst = #mrst{` into its own variable so it would fit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
nickva commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r632074634
##########
File path: src/couch_views/test/couch_views_indexer_test.erl
##########
@@ -610,6 +613,23 @@ index_can_recover_from_crash(Db) ->
], Out).
+handle_acquire_map_context_error(_) ->
+ meck:new(mock_language_server, [non_strict]),
+ config:set("couch_eval.languages", ?QUERY_SERVER_LANG_STRING,
+ atom_to_list(mock_language_server)),
+ meck:expect(mock_language_server, acquire_map_context,
+ fun(_) -> {error, foo_error} end),
+ {'EXIT', {Reason, _}} = (catch couch_views_indexer:start_query_server(#mrst{
Review comment:
I think `?assertError(foo_error, ` might work better here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a change in pull request #3553: Add error propagation to couch_eval and couch_views_indexer
Posted by GitBox <gi...@apache.org>.
nickva commented on a change in pull request #3553:
URL: https://github.com/apache/couchdb/pull/3553#discussion_r631313894
##########
File path: src/couch_views/src/couch_views_indexer.erl
##########
@@ -561,15 +561,19 @@ start_query_server(#mrst{qserver = nil} = Mrst) ->
lib = Lib,
views = Views
} = Mrst,
- {ok, QServer} = couch_eval:acquire_map_context(
+ case couch_eval:acquire_map_context(
DbName,
DDocId,
Language,
Sig,
Lib,
[View#mrview.def || View <- Views]
- ),
- Mrst#mrst{qserver = QServer};
+ ) of
+ {ok, QServer} ->
+ Mrst#mrst{qserver = QServer};
+ {error, Error} ->
+ throw(Error)
Review comment:
Let's use error instead of a `throw`, usually a `throw` is used to a more expected early return here it feels more like an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org