Use ExUnit to write unit tests.

[Ilya Khlopotov <ii...@apache.org>]

Hi everyone,

I am not exactly sure how to proceed with an RFC https://github.com/apache/couchdb-documentation/pull/415 about using ExUnit to write unit tests. I am using information from https://couchdb.apache.org/bylaws.html#rfc
- introduction of a new testing framework is a technical decission and doesn't need an RFC
- on the other hand ExUnit makes Elixir dependency mandatory which is something that need to be agreed upon
- also it seems that this thread is not in correct one to discuss an RFC since it doesn't include [DISCUSSION] prefix.

Please advice how to classify introduction of Elixir based testing framework into unit testing. 

Best regards,
iilyak

On 2019/05/22 18:42:03, Ilya Khlopotov <ii...@apache.org> wrote: 
> Hi everyone,
> 
> With the upgrade of supported Erlang version and introduction of Elixir into our integration test suite we have an opportunity to replace currently used eunit (for new tests only) with Elixir based ExUnit. 
> The eunit testing framework is very hard to maintain. In particular, it has the following problems:
> - the process structure is designed in such a way that failure in setup or teardown of one test affects the execution environment of subsequent tests. Which makes it really hard to locate the place where the problem is coming from.
> - inline test in the same module as the functions it tests might be skipped
> - incorrect usage of ?assert vs ?_assert is not detectable since it makes tests pass 
> - there is a weird (and hard to debug) interaction when used in combination with meck 
>    - https://github.com/eproxus/meck/issues/133#issuecomment-113189678
>    - https://github.com/eproxus/meck/issues/61
>    - meck:unload() must be used instead of meck:unload(Module)
> - teardown is not always run, which affects all subsequent tests
> - grouping of tests is tricky
> - it is hard to group tests so individual tests have meaningful descriptions
> 
> We believe that with ExUnit we wouldn't have these problems:
> - on_exit function is reliable in ExUnit
> - it is easy to group tests using `describe` directive
> - code-generation is trivial, which makes it is possible to generate tests from formal spec (if/when we have one)
> 
> Here are a few examples:
> 
> # Test adapters to test different interfaces using same test suite
> 
> CouchDB has four different interfaces which we need to test. These are:
> - chttpd
> - couch_httpd
> - fabric
> - couch_db
> 
> There is a bunch of operations which are very similar. The only differences between them are:
> - setup/teardown needs different set of applications
> - we need to use different modules to test the operations
> 
> This problem is solved by using testing adapter. We would define a common protocol, which we would use for testing.
> Then we implement this protocol for every interface we want to use.
> 
> ```
> defmodule Couch.Test.CRUD do
>   use ExUnit.Case
>   alias Couch.Test.Adapter
>   alias Couch.Test.Utils, as: Utils
> 
>   alias Couch.Test.Setup
> 
>   require Record
> 
>   test_groups = [
>     "using Clustered API": Adapter.Clustered,
>     "using Backdoor API": Adapter.Backdoor,
>     "using Fabric API": Adapter.Fabric,
>   ]
> 
>   for {describe, adapter} <- test_groups do
>     describe "Database CRUD #{describe}" do
>       @describetag setup: %Setup{}
>         |> Setup.Start.new([:chttpd])
>         |> Setup.Adapter.new(adapter)
>         |> Setup.Admin.new(user: "adm", password: "pass")
>         |> Setup.Login.new(user: "adm", password: "pass")
>       test "Create", %{setup: setup} do
>         db_name = Utils.random_name("db")
>         setup_ctx = setup |> Setup.run()
>         assert {:ok, resp} = Adapter.create_db(Setup.get(setup_ctx, :adapter), db_name)
>         assert resp.body["ok"]
>       end
>     end
>   end
> end
> ```
> 
> # Using same test suite to compare new implementation of the same interface with the old one
> 
> Imagine that we are doing a major rewrite of a module which would implement the same interface.
> How do we compare both implementations return the same results for the same input?
> It is easy in Elixir, here is a sketch:
> ```
> defmodule Couch.Test.Fabric.Rewrite do
>   use ExUnit.Case
>   alias Couch.Test.Utils, as: Utils
> 
>   # we cannot use defrecord here because we need to construct
>   # record at compile time
>   admin_ctx = {:user_ctx, Utils.erlang_record(
>     :user_ctx, "couch/include/couch_db.hrl", roles: ["_admin"])}
> 
>   test_cases = [
>     {"create database": {create_db, [:db_name, []]}},
>     {"create database as admin": {create_db, [:db_name, [admin_ctx]]}}
>   ]
>   module_a = :fabric
>   module_b = :fabric3
> 
>   describe "Test compatibility of '#{module_a}' with '#{module_b}'" do
>     for {description, {function, args}} <- test_cases do
>       test "#{description}" do
>         result_a = unquote(module_a).unquote(function)(unquote_splicing(args))
>         result_b = unquote(module_b).unquote(function)(unquote_splicing(args))
>         assert result_a == result_b
>       end
>     end
>   end
> 
> end
> ```
> As a result we would get following tests
> ```
> Couch.Test.Fabric.Rewrite
>   * test Test compatibility of 'fabric' with 'fabric3' create database (0.01ms)
>   * test Test compatibility of 'fabric' with 'fabric3' create database as admin (0.01ms)
> ```
> 
> The prototype of integration is in this draft PR https://github.com/apache/couchdb/pull/2036. I am planing to write formal RFC after first round of discussions on ML.
> 
> Best regards,
> iilyak
>