You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@couchdb.apache.org by ja...@apache.org on 2021/05/30 01:13:18 UTC

[couchdb] branch sm60-darwin-invalid-offset created (now a3f6313)

This is an automated email from the ASF dual-hosted git repository.

jaydoane pushed a change to branch sm60-darwin-invalid-offset
in repository https://gitbox.apache.org/repos/asf/couchdb.git.


      at a3f6313  Implement aegis:key_manager/0

This branch includes the following new commits:

     new bc3ae96  Do not warn invalid offset for spidermonkey 60 on mac
     new e3d6911  Handle missing db encryption key from lookup
     new 2f617e8  Fix aegis get_db_info server test for operational key manager
     new a3f6313  Implement aegis:key_manager/0

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


[couchdb] 04/04: Implement aegis:key_manager/0

Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

jaydoane pushed a commit to branch sm60-darwin-invalid-offset
in repository https://gitbox.apache.org/repos/asf/couchdb.git

commit a3f6313b5799338c4322aef2fbe2bbb819651b55
Author: Jay Doane <ja...@apache.org>
AuthorDate: Sat May 29 18:09:35 2021 -0700

    Implement aegis:key_manager/0
    
    A convenience function for configuration debugging.
---
 src/aegis/src/aegis.erl | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/aegis/src/aegis.erl b/src/aegis/src/aegis.erl
index d3f7f96..f2da69f 100644
--- a/src/aegis/src/aegis.erl
+++ b/src/aegis/src/aegis.erl
@@ -18,6 +18,7 @@
 
 
 -export([
+    key_manager/0,
     init_db/2,
     open_db/1,
     get_db_info/1,
@@ -28,6 +29,10 @@
     wrap_fold_fun/2
 ]).
 
+key_manager() ->
+    ?AEGIS_KEY_MANAGER.
+
+
 init_db(#{} = Db, Options) ->
     Db#{
         is_encrypted => aegis_server:init_db(Db, Options)

[couchdb] 02/04: Handle missing db encryption key from lookup

Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

jaydoane pushed a commit to branch sm60-darwin-invalid-offset
in repository https://gitbox.apache.org/repos/asf/couchdb.git

commit e3d6911dd386141b934337af0c1ed270a7198580
Author: Jay Doane <ja...@apache.org>
AuthorDate: Thu May 27 22:17:15 2021 -0700

    Handle missing db encryption key from lookup
    
    While performance testing, stack traces were caused by a `badmatch` error
    in aegis_server:
    
    ```
    May 6 21:34:48 c-fdbcore-perf-api-5bc54ff569-jkjxx db error [error] 2021-05-06T21:34:41.337533Z dbcore@127.0.0.1 <0.18553.273> 6adfd54aa5 req_err(3074846293) {{badmatch,{error,not_found}},
     [{aegis_server,handle_call,3,[{file,"src/aegis_server.erl"},{line,170}]},
      {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,636}]},
      {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,665}]},
      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} : {gen_server,call,
        [aegis_server,
         {encrypt,
             #{after_doc_read => undefined,before_doc_update => undefined,
               check_current_ts => -576456460633,db_options => [],
               db_prefix => <<21,22,21,15,1,23,5,202,10,0>>,
               db_version => <<"f518950b94a759bb410f86d32126f421">>,
               interactive => true,is_encrypted => true,
               layer_prefix => <<21,22>>,
               md_version => <<0,0,12,155,121,36,204,80,0,0>>,
               name =>
                   <<"test-user-standard-0002/ccm_capacity_unit_1620334990_test-user-standard-1512">>,
               revs_limit => 1000,
               security_doc => {[]},
               security_fun => undefined,
               tx => {erlfdb_transaction,#Ref<0.4001737934.1270480897.28885>},
               user_ctx =>
                   {user_ctx,<<"test-user-standard-0002">>,
                       [<<"_admin">>,<<"_reader">>,<<"_writer">>],
                       <<"cookie">>},
               uuid => <<"97e681cfd7224680f72213b8bf59a25c">>,
               validate_doc_update_funs => []},
    
        [<<"gen_server:call/2 L206">>,<<"aegis_server:encrypt/3 L93">>,<<"fabric2_fdb:-write_doc_body/2-fun-0-/3 L1430">>,<<"lists:foreach/2 L1338">>,<<"fabric2_fdb:write_doc/6 L851">>,<<"fabric2_db:update_doc_interactive/4 L1944">>,<<"fabric2_db:update_docs_interactive/5 L1808">>,<<"fabric2_db:batch_update_interactive_tx/1 L1752">>]
    ```
    
    This crash is occurrs in aegis_server [1] (now at a slightly different
    line from the trace) because `lookup/2` returns `{error, not_found}`
    [2] when unable to find the UUID in the cache.
    
    This handles the error case, preventing a crash, and also DRYs out the
    code by factoring a `handle_crypto_call/6` function for both
    encryption operations.
    
    [1] https://github.com/apache/couchdb/blob/main/src/aegis/src/aegis_server.erl#L173
    [2] https://github.com/apache/couchdb/blob/main/src/aegis/src/aegis_server.erl#L339
---
 src/aegis/src/aegis_server.erl | 64 ++++++++++++++++++------------------------
 1 file changed, 28 insertions(+), 36 deletions(-)

diff --git a/src/aegis/src/aegis_server.erl b/src/aegis/src/aegis_server.erl
index 15fea4c..aebd528 100644
--- a/src/aegis/src/aegis_server.erl
+++ b/src/aegis/src/aegis_server.erl
@@ -165,43 +165,11 @@ handle_call({insert_key, UUID, DbKey}, _From, #{cache := Cache} = St) ->
     NewSt = insert(St, UUID, DbKey),
     {reply, ok, NewSt, ?TIMEOUT};
 
-handle_call({encrypt, #{uuid := UUID} = Db, Key, Value}, From, St) ->
-
-    {ok, DbKey} = lookup(St, UUID),
-
-    erlang:spawn(fun() ->
-        process_flag(sensitive, true),
-        try
-            do_encrypt(DbKey, Db, Key, Value)
-        of
-            Resp ->
-                gen_server:reply(From, Resp)
-        catch
-            _:Error ->
-                gen_server:reply(From, {error, Error})
-        end
-    end),
-
-    {noreply, St, ?TIMEOUT};
-
-handle_call({decrypt, #{uuid := UUID} = Db, Key, Value}, From, St) ->
-
-    {ok, DbKey} = lookup(St, UUID),
-
-    erlang:spawn(fun() ->
-        process_flag(sensitive, true),
-        try
-            do_decrypt(DbKey, Db, Key, Value)
-        of
-            Resp ->
-                gen_server:reply(From, Resp)
-        catch
-            _:Error ->
-                gen_server:reply(From, {error, Error})
-        end
-    end),
+handle_call({encrypt, Db, Key, Value}, From, St) ->
+    handle_crypto_call(fun do_encrypt/4, Db, Key, Value, From, St);
 
-    {noreply, St, ?TIMEOUT};
+handle_call({decrypt, Db, Key, Value}, From, St) ->
+    handle_crypto_call(fun do_decrypt/4, Db, Key, Value, From, St);
 
 handle_call(_Msg, _From, St) ->
     {noreply, St}.
@@ -233,6 +201,30 @@ code_change(_OldVsn, St, _Extra) ->
 
 %% private functions
 
+
+handle_crypto_call(DoCryptoOp, Db, Key, Value, From, St) ->
+    #{uuid := UUID} = Db,
+    couch_log:info("handle_crypto_call ~p ~p ~p", [DoCryptoOp, Db, Key]),
+    case lookup(St, UUID) of
+        {error, not_found} ->
+            gen_server:reply(From, {error, {db_key_not_found_for, UUID}});
+        {ok, DbKey} ->
+            erlang:spawn(fun() ->
+                process_flag(sensitive, true),
+                try
+                    DoCryptoOp(DbKey, Db, Key, Value)
+                of
+                    Resp ->
+                        gen_server:reply(From, Resp)
+                catch
+                    _:Error ->
+                        gen_server:reply(From, {error, Error})
+                end
+            end),
+            {noreply, St, ?TIMEOUT}
+    end.
+
+
 do_open_db(#{uuid := UUID} = Db) ->
     case ?AEGIS_KEY_MANAGER:open_db(Db) of
         {ok, DbKey} ->

[couchdb] 01/04: Do not warn invalid offset for spidermonkey 60 on mac

Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

jaydoane pushed a commit to branch sm60-darwin-invalid-offset
in repository https://gitbox.apache.org/repos/asf/couchdb.git

commit bc3ae96fd24f937b31627bb21986093f78a42bb1
Author: Jay Doane <ja...@apache.org>
AuthorDate: Wed May 26 22:31:54 2021 -0700

    Do not warn invalid offset for spidermonkey 60 on mac
---
 src/couch/rebar.config.script | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/couch/rebar.config.script b/src/couch/rebar.config.script
index 721c813..96a0489 100644
--- a/src/couch/rebar.config.script
+++ b/src/couch/rebar.config.script
@@ -119,7 +119,7 @@ end.
         };
     {unix, darwin} when SMVsn == "60" ->
         {
-            "-DXP_UNIX -I/usr/include/mozjs-60 -I/usr/local/include/mozjs-60 -std=c++14",
+            "-DXP_UNIX -I/usr/include/mozjs-60 -I/usr/local/include/mozjs-60 -std=c++14 -Wno-invalid-offsetof",
             "-L/usr/local/lib -lmozjs-60 -lm -std=c++14 -lc++"
         };
     {unix, _} when SMVsn == "60" ->

[couchdb] 03/04: Fix aegis get_db_info server test for operational key manager

Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

jaydoane pushed a commit to branch sm60-darwin-invalid-offset
in repository https://gitbox.apache.org/repos/asf/couchdb.git

commit 2f617e8e6b58b76e4e97a669936e4313d70280c6
Author: Jay Doane <ja...@apache.org>
AuthorDate: Sat May 29 16:46:53 2021 -0700

    Fix aegis get_db_info server test for operational key manager
    
    When aegis is configured to use an operational key manager, the
    get_db_info test can fail since the `passthrough` option is used, but
    `get_db_info/1` doesn't get mocked, resulting in the operational key
    manager's implementation unintentially being used in the test.
    
    This replaces the `passthrough` option with `non_strict`, and mocks
    `get_db_info/1` so that the same mocked functions are used in tests
    regardless of whether `AEGIS_KEY_MANAGER` is `aegis_noop_key_manager`,
    or an operational key manager.
---
 src/aegis/rebar.config.script        | 1 +
 src/aegis/test/aegis_server_test.erl | 4 +++-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/src/aegis/rebar.config.script b/src/aegis/rebar.config.script
index ef148bf..a72f27a 100644
--- a/src/aegis/rebar.config.script
+++ b/src/aegis/rebar.config.script
@@ -25,6 +25,7 @@ AegisKeyManager = case lists:keyfind(aegis_key_manager, 1, CouchConfig) of
     _ ->
         aegis_noop_key_manager
 end,
+io:format("AegisKeyManager = ~p~n", [AegisKeyManager]),
 
 CurrentOpts = case lists:keyfind(erl_opts, 1, CONFIG) of
     {erl_opts, Opts} -> Opts;
diff --git a/src/aegis/test/aegis_server_test.erl b/src/aegis/test/aegis_server_test.erl
index ff16475..8dcf01c 100644
--- a/src/aegis/test/aegis_server_test.erl
+++ b/src/aegis/test/aegis_server_test.erl
@@ -50,10 +50,12 @@ basic_test_() ->
 
 
 setup() ->
+    ?debugVal(?AEGIS_KEY_MANAGER),
     Ctx = test_util:start_couch([fabric]),
-    meck:new([?AEGIS_KEY_MANAGER], [passthrough]),
+    meck:new([?AEGIS_KEY_MANAGER], [non_strict]),
     ok = meck:expect(?AEGIS_KEY_MANAGER, init_db, 2, {ok, <<0:256>>}),
     ok = meck:expect(?AEGIS_KEY_MANAGER, open_db, 1, {ok, <<0:256>>}),
+    ok = meck:expect(?AEGIS_KEY_MANAGER, get_db_info, 1, []),
     Ctx.