You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Ilya Khlopotov <ii...@apache.org> on 2019/08/15 14:09:03 UTC
[PROPOSAL] Gradually replace rebar with mix
Hello,
There is an https://github.com/apache/couchdb/issues/1428 issue about migrating to rebar3 or mix. I did an experiment to switch from rebar into mix and wanted to share the results. The code for experiment is here https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. Overall I am happy with the experiment.
# Requirements
1. Should be able to compile dependencies which use either rebar or rebar3
2. Should be able to lock dependencies to specific versions using references to commits
3. Should be able to call eunit test suite
4. Should be able to compile NIFs
5. Should be easy to install plugins
6. Should support dialyzer
7. Should support release managers
8. Support for raw dependencies
9. Produce source tarball suitable for offline compilation (ASF requirement)
# How mix supports the requirements
1. Mix is able to figure out which build tool to use. There is also a possibility to specify build tool individually for a given dependency.
`manager: mix | rebar | rebar3 | make`. There is also support for calling arbitrary shell command configured via `compile: <command>`. The manual step is required to install the needed tools `mix local.rebar --force`.
However there are means to automate that step (we can use Makefile or define a `bootstrap` mix alias )
2. Dependencies are locked via `mix.lock` file.
3. Eunit is supported via one of the mix plugins. Both plugins are very old but it should be easy to write our own (32 lines of code).
- https://github.com/dantswain/mix_eunit
- https://github.com/talentdeficit/mixunit
4. NIFs can be compiled using one of the following approaches:
- specify `rebar` as build manager for a dependency (easiest)
- call a Makefile from mix.exs. See examples:
- https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
- https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
- use https://github.com/davisp/erlang-native-compiler
- use https://github.com/blt/port_compiler
5. Installing plugins for mix is similar to rebar3. It is sufficient to just specify list of plugins in the deps in mix.exs
6. There are multiple dyalizer plugins to choose from
- https://github.com/jeremyjh/dialyxir
- https://github.com/gabrielgatu/mix-dialyzer
7. Multiple release managers are supported via mix plugins
- distillery is supported via https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
- relx and reltools can be used via https://github.com/bitwalker/exrm (deprecated in favor of distillery)
- we can also roll our own mix alias using https://github.com/okeuday/reltool_util
- latest elixir versions have `release` command out of the box
8. Raw dependencies are supported via `compile: false` option
9. I think it is doable. In the worst case scenario we could just tar the content of `src` directory after calling `mix deps.get`
# Warts of mix in the context of CouchDB project
- CouchDB doesn't use standard elixir directories layout which caused include_lib directives to not work. To solve this issue absolute path to `<root>/src` need to be added into include path
- for dependencies managed by mix it is done via `erlc_options` parameter
- Unfortunately when we call out to rebar there is no way to modify arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS` environment variable. Which is not very elegant solution.
- We need to create empty `rebar.config` file for erlang applications which do not have `rebar.config` (all applications in couchdb repo). This problem can be fixed by adding mix.exs to dependencies we have control over.
- The mix is slow when we have to call out to rebar. This problem can be fixed by adding mix.exs to dependencies we have control over.
# Roadmap
The full conversion would take some time and if we decide to go with mix it makes sense to do it incrementally. Possible roadmap could be:
- initial PR
- offload dependency fetching to mix.exs and generate rebar.config so release machinery still works
- Update Makefile targets to call mix
- Choose or implement eunit plugin for mix
- Support running tests individually (replacement for `make eunit apps= tests= suites=`)
- Replace rebar plugins we use with alternative solutions
- Figure out how to use dialyzer
- Figure out how to use mix for preparing release
Best regards,
ILYA (aka iilyak)
Re: [PROPOSAL] Gradually replace rebar with mix
Posted by Joan Touzet <wo...@apache.org>.
Jenkins driven Windows CI is in the cards. We now have Azure credits we
can put towards this, or perhaps one of the workers IBM is donating can
run this.
It'll have to be a "snowflake" machine build, but I believe it can happen.
But you should be able to run a Windows VM using VirtualBox? And we can
get you a Windows license if that's an issue.
It's kinda late to be solving it in CI...
-Joan
On 2019-08-20 8:56, Ilya Khlopotov wrote:
> I am afraid that until we have a Jenkins driven Windows CI we wouldn't be able to modernize any of the build or release plumbing.
>
> Best regards,
> iilyak
>
> On 2019/08/20 12:28:12, Garren Smith <ga...@apache.org> wrote:
>> I'm a big +1 on this. Initially, I know we would have some teething issues
>> getting this working. But I don't think we can stay on rebar2 forever
>> either. Adding in erlang dependancies is becoming increasingly more
>> difficult as they expect rebar3.
>>
>> Cheers
>> Garren
>>
>> On Fri, Aug 16, 2019 at 2:02 PM Ilya Khlopotov <ii...@apache.org> wrote:
>>
>>>
>>>
>>> On 2019/08/15 18:51:29, Joan Touzet <wo...@apache.org> wrote:
>>>> Hi Ilya,
>>>>
>>>>
>>>> On 2019-08-15 10:09, Ilya Khlopotov wrote:
>>>>> Hello,
>>>>>
>>>>> There is an https://github.com/apache/couchdb/issues/1428 issue about
>>> migrating to rebar3 or mix. I did an experiment to switch from rebar into
>>> mix and wanted to share the results. The code for experiment is here
>>> https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix.
>>> Overall I am happy with the experiment.
>>>>>
>>>>> # Requirements
>>>>>
>>>>> 1. Should be able to compile dependencies which use either rebar or
>>> rebar3
>>>>> 2. Should be able to lock dependencies to specific versions using
>>> references to commits
>>>>> 3. Should be able to call eunit test suite
>>>>> 4. Should be able to compile NIFs
>>>>> 5. Should be easy to install plugins
>>>>> 6. Should support dialyzer
>>>>> 7. Should support release managers
>>>>> 8. Support for raw dependencies
>>>>> 9. Produce source tarball suitable for offline compilation (ASF
>>> requirement)
>>>>>
>>>>> # How mix supports the requirements
>>>>>
>>>>> 1. Mix is able to figure out which build tool to use. There is also a
>>> possibility to specify build tool individually for a given dependency.
>>>>> `manager: mix | rebar | rebar3 | make`. There is also support for
>>> calling arbitrary shell command configured via `compile: <command>`. The
>>> manual step is required to install the needed tools `mix local.rebar
>>> --force`.
>>>>> However there are means to automate that step (we can use Makefile
>>> or define a `bootstrap` mix alias )
>>>>> 2. Dependencies are locked via `mix.lock` file.
>>>>
>>>> I don't have any strong opinions one way or the other, but I did see
>>>> Garren asking about 'hex' instead of / in conjunction with 'mix' for
>>>> dependency management. Is this effort combined with his or independent?
>>>> I don't know the tools well enough.
>>>
>>> mix works very well with hex
>>>
>>>>
>>>>> 3. Eunit is supported via one of the mix plugins. Both plugins are
>>> very old but it should be easy to write our own (32 lines of code).
>>>>> - https://github.com/dantswain/mix_eunit
>>>>> - https://github.com/talentdeficit/mixunit
>>>>> 4. NIFs can be compiled using one of the following approaches:
>>>>> - specify `rebar` as build manager for a dependency (easiest)
>>>>
>>>> Are you considering staying with rebar2 here, or moving to rebar(3)? It
>>>> would be nice not to have to keep supporting (and shipping!) our fork of
>>>> rebar2.
>>> I considered to stay with rebar2 for first iteration. Could you remind me
>>> why we are forking rebar?
>>> I did check that our version has no differences from upstream.
>>> git remote -v
>>> origin https://github.com/apache/couchdb-rebar (fetch)
>>> origin https://github.com/apache/couchdb-rebar (push)
>>> upstream https://github.com/rebar/rebar (fetch)
>>> upstream https://github.com/rebar/rebar (push)
>>>
>>> git log --oneline origin/master | head
>>> 0c435f3 Update master with mainline rebar (v2)
>>> b6d3094 Merge pull request #620 from tuncer/travis-dialyze
>>> 8a2aca0 travis-ci: allow Dialyzer job to fail
>>> 16d5dfe travis-ci: update otp versions
>>> 4e372a3 travis-ci: enable 20.1
>>> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning
>>> b05882a rebar_cover: ignore opaque type match Dialyzer warning
>>> 903c89d rebar_utils: fix Dialyzer warning
>>> b36e72b Run Dialyzer on Travis-CI
>>> 4bd43fe Merge pull request #645 from Juliusan/log_fix
>>>
>>> git log --oneline upstream/master | head
>>> b6d3094 Merge pull request #620 from tuncer/travis-dialyze
>>> 8a2aca0 travis-ci: allow Dialyzer job to fail
>>> 16d5dfe travis-ci: update otp versions
>>> 4e372a3 travis-ci: enable 20.1
>>> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning
>>> b05882a rebar_cover: ignore opaque type match Dialyzer warning
>>> 903c89d rebar_utils: fix Dialyzer warning
>>> b36e72b Run Dialyzer on Travis-CI
>>> 4bd43fe Merge pull request #645 from Juliusan/log_fix
>>> 2276f85 Log out success message with newlines
>>>
>>> git diff origin/master upstream/master
>>>
>>>>
>>>>> - call a Makefile from mix.exs. See examples:
>>>>> - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
>>>>> - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
>>>>> - use https://github.com/davisp/erlang-native-compiler
>>>>> - use https://github.com/blt/port_compiler
>>>>
>>>> Paul left a comment about this on the roadmap issue just today. Can you
>>>> respond to his comment (it was related tot dependencies-of-dependencies,
>>>> i.e. jiffy)
>>>>
>>>> He also raised the concern that it may force us to use only the very
>>>> latest Erlang versions - can you comment on this? Does the current
>>>> hex/mix situation require a minimum version? What version is that?
>>>
>>> I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported
>>> on erlang 19 which is AFAIK a minimal erlang version CouchDB supports.
>>>
>>>>> 5. Installing plugins for mix is similar to rebar3. It is sufficient
>>> to just specify list of plugins in the deps in mix.exs
>>>>> 6. There are multiple dyalizer plugins to choose from
>>>>> - https://github.com/jeremyjh/dialyxir
>>>>> - https://github.com/gabrielgatu/mix-dialyzer
>>>>
>>>> I know this is a requirement for Cloudant, but today isn't for base
>>>> CouchDB. I would prefer not to introduce this requirement at the same
>>>> time as migrating build tool.
>>>
>>> The reason I mentioned it here is because we have 'make dialyze' command.
>>>
>>>>> 7. Multiple release managers are supported via mix plugins
>>>>> - distillery is supported via
>>> https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
>>>>> - relx and reltools can be used via
>>> https://github.com/bitwalker/exrm (deprecated in favor of distillery)
>>>>> - we can also roll our own mix alias using
>>> https://github.com/okeuday/reltool_util
>>>>> - latest elixir versions have `release` command out of the box
>>>>> 8. Raw dependencies are supported via `compile: false` option
>>>>> 9. I think it is doable. In the worst case scenario we could just tar
>>> the content of `src` directory after calling `mix deps.get`
>>>>>
>>>>> # Warts of mix in the context of CouchDB project
>>>>>
>>>>> - CouchDB doesn't use standard elixir directories layout which caused
>>> include_lib directives to not work. To solve this issue absolute path to
>>> `<root>/src` need to be added into include path
>>>>> - for dependencies managed by mix it is done via `erlc_options`
>>> parameter
>>>>> - Unfortunately when we call out to rebar there is no way to modify
>>> arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS`
>>> environment variable. Which is not very elegant solution.
>>>>
>>>> This may also cause problems on Windows, where we have very specific
>>>> requirements for compiler options so as to ensure the correct libraries
>>>> are linked for couchjs and so on.
>>>>
>>>> Have you tried compiling on that platform? ;) (Please don't make me do
>>>> all the work here, again...)
>>>>
>>>>> - We need to create empty `rebar.config` file for erlang applications
>>> which do not have `rebar.config` (all applications in couchdb repo). This
>>> problem can be fixed by adding mix.exs to dependencies we have control over.
>>>>> - The mix is slow when we have to call out to rebar. This problem can
>>> be fixed by adding mix.exs to dependencies we have control over.
>>>>
>>>> How much slower? Can you post a comparison of `time make couch` between
>>>> master and your branch?
>>>
>>> master:
>>> > time bin/rebar get-deps update-deps
>>> real 0m37.514s
>>> user 0m1.960s
>>> sys 0m3.300s
>>> > time make couch
>>> real 0m49.677s
>>> user 0m46.910s
>>> sys 0m22.210s
>>>
>>> mix:
>>> > time mix do deps.get
>>> real 0m24.405s
>>> user 0m1.590s
>>> sys 0m1.270s
>>> > time mix deps.compile
>>> real 3m47.968s
>>> user 0m0.860s
>>> sys 0m0.640s
>>>
>>>>> # Roadmap
>>>>>
>>>>> The full conversion would take some time and if we decide to go with
>>> mix it makes sense to do it incrementally. Possible roadmap could be:
>>>>> - initial PR
>>>>> - offload dependency fetching to mix.exs and generate rebar.config
>>> so release machinery still works
>>>>> - Update Makefile targets to call mix
>>>>> - Choose or implement eunit plugin for mix
>>>>> - Support running tests individually (replacement for `make eunit
>>> apps= tests= suites=`)
>>>>> - Replace rebar plugins we use with alternative solutions
>>>>> - Figure out how to use dialyzer
>>>>> - Figure out how to use mix for preparing release
>>>>
>>>> Anything that lands on master must support release preparation. If your
>>>> incremental approach breaks `make dist`, you'll need to stage all of
>>>> your incremental changes on a feature branch and fix `make dist` before
>>>> merging to master. master needs to be releasable at all times.
>>> +1000
>>>
>>>>> Best regards,
>>>>> ILYA (aka iilyak)
>>>>
>>>> -Joan "release engineer hat is now off" Touzet
>>>>
>>>>
>>>
>>
Re: [PROPOSAL] Gradually replace rebar with mix
Posted by Ilya Khlopotov <ii...@apache.org>.
I am afraid that until we have a Jenkins driven Windows CI we wouldn't be able to modernize any of the build or release plumbing.
Best regards,
iilyak
On 2019/08/20 12:28:12, Garren Smith <ga...@apache.org> wrote:
> I'm a big +1 on this. Initially, I know we would have some teething issues
> getting this working. But I don't think we can stay on rebar2 forever
> either. Adding in erlang dependancies is becoming increasingly more
> difficult as they expect rebar3.
>
> Cheers
> Garren
>
> On Fri, Aug 16, 2019 at 2:02 PM Ilya Khlopotov <ii...@apache.org> wrote:
>
> >
> >
> > On 2019/08/15 18:51:29, Joan Touzet <wo...@apache.org> wrote:
> > > Hi Ilya,
> > >
> > >
> > > On 2019-08-15 10:09, Ilya Khlopotov wrote:
> > > > Hello,
> > > >
> > > > There is an https://github.com/apache/couchdb/issues/1428 issue about
> > migrating to rebar3 or mix. I did an experiment to switch from rebar into
> > mix and wanted to share the results. The code for experiment is here
> > https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix.
> > Overall I am happy with the experiment.
> > > >
> > > > # Requirements
> > > >
> > > > 1. Should be able to compile dependencies which use either rebar or
> > rebar3
> > > > 2. Should be able to lock dependencies to specific versions using
> > references to commits
> > > > 3. Should be able to call eunit test suite
> > > > 4. Should be able to compile NIFs
> > > > 5. Should be easy to install plugins
> > > > 6. Should support dialyzer
> > > > 7. Should support release managers
> > > > 8. Support for raw dependencies
> > > > 9. Produce source tarball suitable for offline compilation (ASF
> > requirement)
> > > >
> > > > # How mix supports the requirements
> > > >
> > > > 1. Mix is able to figure out which build tool to use. There is also a
> > possibility to specify build tool individually for a given dependency.
> > > > `manager: mix | rebar | rebar3 | make`. There is also support for
> > calling arbitrary shell command configured via `compile: <command>`. The
> > manual step is required to install the needed tools `mix local.rebar
> > --force`.
> > > > However there are means to automate that step (we can use Makefile
> > or define a `bootstrap` mix alias )
> > > > 2. Dependencies are locked via `mix.lock` file.
> > >
> > > I don't have any strong opinions one way or the other, but I did see
> > > Garren asking about 'hex' instead of / in conjunction with 'mix' for
> > > dependency management. Is this effort combined with his or independent?
> > > I don't know the tools well enough.
> >
> > mix works very well with hex
> >
> > >
> > > > 3. Eunit is supported via one of the mix plugins. Both plugins are
> > very old but it should be easy to write our own (32 lines of code).
> > > > - https://github.com/dantswain/mix_eunit
> > > > - https://github.com/talentdeficit/mixunit
> > > > 4. NIFs can be compiled using one of the following approaches:
> > > > - specify `rebar` as build manager for a dependency (easiest)
> > >
> > > Are you considering staying with rebar2 here, or moving to rebar(3)? It
> > > would be nice not to have to keep supporting (and shipping!) our fork of
> > > rebar2.
> > I considered to stay with rebar2 for first iteration. Could you remind me
> > why we are forking rebar?
> > I did check that our version has no differences from upstream.
> > git remote -v
> > origin https://github.com/apache/couchdb-rebar (fetch)
> > origin https://github.com/apache/couchdb-rebar (push)
> > upstream https://github.com/rebar/rebar (fetch)
> > upstream https://github.com/rebar/rebar (push)
> >
> > git log --oneline origin/master | head
> > 0c435f3 Update master with mainline rebar (v2)
> > b6d3094 Merge pull request #620 from tuncer/travis-dialyze
> > 8a2aca0 travis-ci: allow Dialyzer job to fail
> > 16d5dfe travis-ci: update otp versions
> > 4e372a3 travis-ci: enable 20.1
> > 73feb5f rebar_xref: ignore opaque type match Dialyzer warning
> > b05882a rebar_cover: ignore opaque type match Dialyzer warning
> > 903c89d rebar_utils: fix Dialyzer warning
> > b36e72b Run Dialyzer on Travis-CI
> > 4bd43fe Merge pull request #645 from Juliusan/log_fix
> >
> > git log --oneline upstream/master | head
> > b6d3094 Merge pull request #620 from tuncer/travis-dialyze
> > 8a2aca0 travis-ci: allow Dialyzer job to fail
> > 16d5dfe travis-ci: update otp versions
> > 4e372a3 travis-ci: enable 20.1
> > 73feb5f rebar_xref: ignore opaque type match Dialyzer warning
> > b05882a rebar_cover: ignore opaque type match Dialyzer warning
> > 903c89d rebar_utils: fix Dialyzer warning
> > b36e72b Run Dialyzer on Travis-CI
> > 4bd43fe Merge pull request #645 from Juliusan/log_fix
> > 2276f85 Log out success message with newlines
> >
> > git diff origin/master upstream/master
> >
> > >
> > > > - call a Makefile from mix.exs. See examples:
> > > > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
> > > > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
> > > > - use https://github.com/davisp/erlang-native-compiler
> > > > - use https://github.com/blt/port_compiler
> > >
> > > Paul left a comment about this on the roadmap issue just today. Can you
> > > respond to his comment (it was related tot dependencies-of-dependencies,
> > > i.e. jiffy)
> > >
> > > He also raised the concern that it may force us to use only the very
> > > latest Erlang versions - can you comment on this? Does the current
> > > hex/mix situation require a minimum version? What version is that?
> >
> > I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported
> > on erlang 19 which is AFAIK a minimal erlang version CouchDB supports.
> >
> > > > 5. Installing plugins for mix is similar to rebar3. It is sufficient
> > to just specify list of plugins in the deps in mix.exs
> > > > 6. There are multiple dyalizer plugins to choose from
> > > > - https://github.com/jeremyjh/dialyxir
> > > > - https://github.com/gabrielgatu/mix-dialyzer
> > >
> > > I know this is a requirement for Cloudant, but today isn't for base
> > > CouchDB. I would prefer not to introduce this requirement at the same
> > > time as migrating build tool.
> >
> > The reason I mentioned it here is because we have 'make dialyze' command.
> >
> > > > 7. Multiple release managers are supported via mix plugins
> > > > - distillery is supported via
> > https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
> > > > - relx and reltools can be used via
> > https://github.com/bitwalker/exrm (deprecated in favor of distillery)
> > > > - we can also roll our own mix alias using
> > https://github.com/okeuday/reltool_util
> > > > - latest elixir versions have `release` command out of the box
> > > > 8. Raw dependencies are supported via `compile: false` option
> > > > 9. I think it is doable. In the worst case scenario we could just tar
> > the content of `src` directory after calling `mix deps.get`
> > > >
> > > > # Warts of mix in the context of CouchDB project
> > > >
> > > > - CouchDB doesn't use standard elixir directories layout which caused
> > include_lib directives to not work. To solve this issue absolute path to
> > `<root>/src` need to be added into include path
> > > > - for dependencies managed by mix it is done via `erlc_options`
> > parameter
> > > > - Unfortunately when we call out to rebar there is no way to modify
> > arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS`
> > environment variable. Which is not very elegant solution.
> > >
> > > This may also cause problems on Windows, where we have very specific
> > > requirements for compiler options so as to ensure the correct libraries
> > > are linked for couchjs and so on.
> > >
> > > Have you tried compiling on that platform? ;) (Please don't make me do
> > > all the work here, again...)
> > >
> > > > - We need to create empty `rebar.config` file for erlang applications
> > which do not have `rebar.config` (all applications in couchdb repo). This
> > problem can be fixed by adding mix.exs to dependencies we have control over.
> > > > - The mix is slow when we have to call out to rebar. This problem can
> > be fixed by adding mix.exs to dependencies we have control over.
> > >
> > > How much slower? Can you post a comparison of `time make couch` between
> > > master and your branch?
> >
> > master:
> > > time bin/rebar get-deps update-deps
> > real 0m37.514s
> > user 0m1.960s
> > sys 0m3.300s
> > > time make couch
> > real 0m49.677s
> > user 0m46.910s
> > sys 0m22.210s
> >
> > mix:
> > > time mix do deps.get
> > real 0m24.405s
> > user 0m1.590s
> > sys 0m1.270s
> > > time mix deps.compile
> > real 3m47.968s
> > user 0m0.860s
> > sys 0m0.640s
> >
> > > > # Roadmap
> > > >
> > > > The full conversion would take some time and if we decide to go with
> > mix it makes sense to do it incrementally. Possible roadmap could be:
> > > > - initial PR
> > > > - offload dependency fetching to mix.exs and generate rebar.config
> > so release machinery still works
> > > > - Update Makefile targets to call mix
> > > > - Choose or implement eunit plugin for mix
> > > > - Support running tests individually (replacement for `make eunit
> > apps= tests= suites=`)
> > > > - Replace rebar plugins we use with alternative solutions
> > > > - Figure out how to use dialyzer
> > > > - Figure out how to use mix for preparing release
> > >
> > > Anything that lands on master must support release preparation. If your
> > > incremental approach breaks `make dist`, you'll need to stage all of
> > > your incremental changes on a feature branch and fix `make dist` before
> > > merging to master. master needs to be releasable at all times.
> > +1000
> >
> > > > Best regards,
> > > > ILYA (aka iilyak)
> > >
> > > -Joan "release engineer hat is now off" Touzet
> > >
> > >
> >
>
Re: [PROPOSAL] Gradually replace rebar with mix
Posted by Garren Smith <ga...@apache.org>.
I'm a big +1 on this. Initially, I know we would have some teething issues
getting this working. But I don't think we can stay on rebar2 forever
either. Adding in erlang dependancies is becoming increasingly more
difficult as they expect rebar3.
Cheers
Garren
On Fri, Aug 16, 2019 at 2:02 PM Ilya Khlopotov <ii...@apache.org> wrote:
>
>
> On 2019/08/15 18:51:29, Joan Touzet <wo...@apache.org> wrote:
> > Hi Ilya,
> >
> >
> > On 2019-08-15 10:09, Ilya Khlopotov wrote:
> > > Hello,
> > >
> > > There is an https://github.com/apache/couchdb/issues/1428 issue about
> migrating to rebar3 or mix. I did an experiment to switch from rebar into
> mix and wanted to share the results. The code for experiment is here
> https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix.
> Overall I am happy with the experiment.
> > >
> > > # Requirements
> > >
> > > 1. Should be able to compile dependencies which use either rebar or
> rebar3
> > > 2. Should be able to lock dependencies to specific versions using
> references to commits
> > > 3. Should be able to call eunit test suite
> > > 4. Should be able to compile NIFs
> > > 5. Should be easy to install plugins
> > > 6. Should support dialyzer
> > > 7. Should support release managers
> > > 8. Support for raw dependencies
> > > 9. Produce source tarball suitable for offline compilation (ASF
> requirement)
> > >
> > > # How mix supports the requirements
> > >
> > > 1. Mix is able to figure out which build tool to use. There is also a
> possibility to specify build tool individually for a given dependency.
> > > `manager: mix | rebar | rebar3 | make`. There is also support for
> calling arbitrary shell command configured via `compile: <command>`. The
> manual step is required to install the needed tools `mix local.rebar
> --force`.
> > > However there are means to automate that step (we can use Makefile
> or define a `bootstrap` mix alias )
> > > 2. Dependencies are locked via `mix.lock` file.
> >
> > I don't have any strong opinions one way or the other, but I did see
> > Garren asking about 'hex' instead of / in conjunction with 'mix' for
> > dependency management. Is this effort combined with his or independent?
> > I don't know the tools well enough.
>
> mix works very well with hex
>
> >
> > > 3. Eunit is supported via one of the mix plugins. Both plugins are
> very old but it should be easy to write our own (32 lines of code).
> > > - https://github.com/dantswain/mix_eunit
> > > - https://github.com/talentdeficit/mixunit
> > > 4. NIFs can be compiled using one of the following approaches:
> > > - specify `rebar` as build manager for a dependency (easiest)
> >
> > Are you considering staying with rebar2 here, or moving to rebar(3)? It
> > would be nice not to have to keep supporting (and shipping!) our fork of
> > rebar2.
> I considered to stay with rebar2 for first iteration. Could you remind me
> why we are forking rebar?
> I did check that our version has no differences from upstream.
> git remote -v
> origin https://github.com/apache/couchdb-rebar (fetch)
> origin https://github.com/apache/couchdb-rebar (push)
> upstream https://github.com/rebar/rebar (fetch)
> upstream https://github.com/rebar/rebar (push)
>
> git log --oneline origin/master | head
> 0c435f3 Update master with mainline rebar (v2)
> b6d3094 Merge pull request #620 from tuncer/travis-dialyze
> 8a2aca0 travis-ci: allow Dialyzer job to fail
> 16d5dfe travis-ci: update otp versions
> 4e372a3 travis-ci: enable 20.1
> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning
> b05882a rebar_cover: ignore opaque type match Dialyzer warning
> 903c89d rebar_utils: fix Dialyzer warning
> b36e72b Run Dialyzer on Travis-CI
> 4bd43fe Merge pull request #645 from Juliusan/log_fix
>
> git log --oneline upstream/master | head
> b6d3094 Merge pull request #620 from tuncer/travis-dialyze
> 8a2aca0 travis-ci: allow Dialyzer job to fail
> 16d5dfe travis-ci: update otp versions
> 4e372a3 travis-ci: enable 20.1
> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning
> b05882a rebar_cover: ignore opaque type match Dialyzer warning
> 903c89d rebar_utils: fix Dialyzer warning
> b36e72b Run Dialyzer on Travis-CI
> 4bd43fe Merge pull request #645 from Juliusan/log_fix
> 2276f85 Log out success message with newlines
>
> git diff origin/master upstream/master
>
> >
> > > - call a Makefile from mix.exs. See examples:
> > > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
> > > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
> > > - use https://github.com/davisp/erlang-native-compiler
> > > - use https://github.com/blt/port_compiler
> >
> > Paul left a comment about this on the roadmap issue just today. Can you
> > respond to his comment (it was related tot dependencies-of-dependencies,
> > i.e. jiffy)
> >
> > He also raised the concern that it may force us to use only the very
> > latest Erlang versions - can you comment on this? Does the current
> > hex/mix situation require a minimum version? What version is that?
>
> I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported
> on erlang 19 which is AFAIK a minimal erlang version CouchDB supports.
>
> > > 5. Installing plugins for mix is similar to rebar3. It is sufficient
> to just specify list of plugins in the deps in mix.exs
> > > 6. There are multiple dyalizer plugins to choose from
> > > - https://github.com/jeremyjh/dialyxir
> > > - https://github.com/gabrielgatu/mix-dialyzer
> >
> > I know this is a requirement for Cloudant, but today isn't for base
> > CouchDB. I would prefer not to introduce this requirement at the same
> > time as migrating build tool.
>
> The reason I mentioned it here is because we have 'make dialyze' command.
>
> > > 7. Multiple release managers are supported via mix plugins
> > > - distillery is supported via
> https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
> > > - relx and reltools can be used via
> https://github.com/bitwalker/exrm (deprecated in favor of distillery)
> > > - we can also roll our own mix alias using
> https://github.com/okeuday/reltool_util
> > > - latest elixir versions have `release` command out of the box
> > > 8. Raw dependencies are supported via `compile: false` option
> > > 9. I think it is doable. In the worst case scenario we could just tar
> the content of `src` directory after calling `mix deps.get`
> > >
> > > # Warts of mix in the context of CouchDB project
> > >
> > > - CouchDB doesn't use standard elixir directories layout which caused
> include_lib directives to not work. To solve this issue absolute path to
> `<root>/src` need to be added into include path
> > > - for dependencies managed by mix it is done via `erlc_options`
> parameter
> > > - Unfortunately when we call out to rebar there is no way to modify
> arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS`
> environment variable. Which is not very elegant solution.
> >
> > This may also cause problems on Windows, where we have very specific
> > requirements for compiler options so as to ensure the correct libraries
> > are linked for couchjs and so on.
> >
> > Have you tried compiling on that platform? ;) (Please don't make me do
> > all the work here, again...)
> >
> > > - We need to create empty `rebar.config` file for erlang applications
> which do not have `rebar.config` (all applications in couchdb repo). This
> problem can be fixed by adding mix.exs to dependencies we have control over.
> > > - The mix is slow when we have to call out to rebar. This problem can
> be fixed by adding mix.exs to dependencies we have control over.
> >
> > How much slower? Can you post a comparison of `time make couch` between
> > master and your branch?
>
> master:
> > time bin/rebar get-deps update-deps
> real 0m37.514s
> user 0m1.960s
> sys 0m3.300s
> > time make couch
> real 0m49.677s
> user 0m46.910s
> sys 0m22.210s
>
> mix:
> > time mix do deps.get
> real 0m24.405s
> user 0m1.590s
> sys 0m1.270s
> > time mix deps.compile
> real 3m47.968s
> user 0m0.860s
> sys 0m0.640s
>
> > > # Roadmap
> > >
> > > The full conversion would take some time and if we decide to go with
> mix it makes sense to do it incrementally. Possible roadmap could be:
> > > - initial PR
> > > - offload dependency fetching to mix.exs and generate rebar.config
> so release machinery still works
> > > - Update Makefile targets to call mix
> > > - Choose or implement eunit plugin for mix
> > > - Support running tests individually (replacement for `make eunit
> apps= tests= suites=`)
> > > - Replace rebar plugins we use with alternative solutions
> > > - Figure out how to use dialyzer
> > > - Figure out how to use mix for preparing release
> >
> > Anything that lands on master must support release preparation. If your
> > incremental approach breaks `make dist`, you'll need to stage all of
> > your incremental changes on a feature branch and fix `make dist` before
> > merging to master. master needs to be releasable at all times.
> +1000
>
> > > Best regards,
> > > ILYA (aka iilyak)
> >
> > -Joan "release engineer hat is now off" Touzet
> >
> >
>
Re: [PROPOSAL] Gradually replace rebar with mix
Posted by Ilya Khlopotov <ii...@apache.org>.
On 2019/08/15 18:51:29, Joan Touzet <wo...@apache.org> wrote:
> Hi Ilya,
>
>
> On 2019-08-15 10:09, Ilya Khlopotov wrote:
> > Hello,
> >
> > There is an https://github.com/apache/couchdb/issues/1428 issue about migrating to rebar3 or mix. I did an experiment to switch from rebar into mix and wanted to share the results. The code for experiment is here https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. Overall I am happy with the experiment.
> >
> > # Requirements
> >
> > 1. Should be able to compile dependencies which use either rebar or rebar3
> > 2. Should be able to lock dependencies to specific versions using references to commits
> > 3. Should be able to call eunit test suite
> > 4. Should be able to compile NIFs
> > 5. Should be easy to install plugins
> > 6. Should support dialyzer
> > 7. Should support release managers
> > 8. Support for raw dependencies
> > 9. Produce source tarball suitable for offline compilation (ASF requirement)
> >
> > # How mix supports the requirements
> >
> > 1. Mix is able to figure out which build tool to use. There is also a possibility to specify build tool individually for a given dependency.
> > `manager: mix | rebar | rebar3 | make`. There is also support for calling arbitrary shell command configured via `compile: <command>`. The manual step is required to install the needed tools `mix local.rebar --force`.
> > However there are means to automate that step (we can use Makefile or define a `bootstrap` mix alias )
> > 2. Dependencies are locked via `mix.lock` file.
>
> I don't have any strong opinions one way or the other, but I did see
> Garren asking about 'hex' instead of / in conjunction with 'mix' for
> dependency management. Is this effort combined with his or independent?
> I don't know the tools well enough.
mix works very well with hex
>
> > 3. Eunit is supported via one of the mix plugins. Both plugins are very old but it should be easy to write our own (32 lines of code).
> > - https://github.com/dantswain/mix_eunit
> > - https://github.com/talentdeficit/mixunit
> > 4. NIFs can be compiled using one of the following approaches:
> > - specify `rebar` as build manager for a dependency (easiest)
>
> Are you considering staying with rebar2 here, or moving to rebar(3)? It
> would be nice not to have to keep supporting (and shipping!) our fork of
> rebar2.
I considered to stay with rebar2 for first iteration. Could you remind me why we are forking rebar?
I did check that our version has no differences from upstream.
git remote -v
origin https://github.com/apache/couchdb-rebar (fetch)
origin https://github.com/apache/couchdb-rebar (push)
upstream https://github.com/rebar/rebar (fetch)
upstream https://github.com/rebar/rebar (push)
git log --oneline origin/master | head
0c435f3 Update master with mainline rebar (v2)
b6d3094 Merge pull request #620 from tuncer/travis-dialyze
8a2aca0 travis-ci: allow Dialyzer job to fail
16d5dfe travis-ci: update otp versions
4e372a3 travis-ci: enable 20.1
73feb5f rebar_xref: ignore opaque type match Dialyzer warning
b05882a rebar_cover: ignore opaque type match Dialyzer warning
903c89d rebar_utils: fix Dialyzer warning
b36e72b Run Dialyzer on Travis-CI
4bd43fe Merge pull request #645 from Juliusan/log_fix
git log --oneline upstream/master | head
b6d3094 Merge pull request #620 from tuncer/travis-dialyze
8a2aca0 travis-ci: allow Dialyzer job to fail
16d5dfe travis-ci: update otp versions
4e372a3 travis-ci: enable 20.1
73feb5f rebar_xref: ignore opaque type match Dialyzer warning
b05882a rebar_cover: ignore opaque type match Dialyzer warning
903c89d rebar_utils: fix Dialyzer warning
b36e72b Run Dialyzer on Travis-CI
4bd43fe Merge pull request #645 from Juliusan/log_fix
2276f85 Log out success message with newlines
git diff origin/master upstream/master
>
> > - call a Makefile from mix.exs. See examples:
> > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
> > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
> > - use https://github.com/davisp/erlang-native-compiler
> > - use https://github.com/blt/port_compiler
>
> Paul left a comment about this on the roadmap issue just today. Can you
> respond to his comment (it was related tot dependencies-of-dependencies,
> i.e. jiffy)
>
> He also raised the concern that it may force us to use only the very
> latest Erlang versions - can you comment on this? Does the current
> hex/mix situation require a minimum version? What version is that?
I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported on erlang 19 which is AFAIK a minimal erlang version CouchDB supports.
> > 5. Installing plugins for mix is similar to rebar3. It is sufficient to just specify list of plugins in the deps in mix.exs
> > 6. There are multiple dyalizer plugins to choose from
> > - https://github.com/jeremyjh/dialyxir
> > - https://github.com/gabrielgatu/mix-dialyzer
>
> I know this is a requirement for Cloudant, but today isn't for base
> CouchDB. I would prefer not to introduce this requirement at the same
> time as migrating build tool.
The reason I mentioned it here is because we have 'make dialyze' command.
> > 7. Multiple release managers are supported via mix plugins
> > - distillery is supported via https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
> > - relx and reltools can be used via https://github.com/bitwalker/exrm (deprecated in favor of distillery)
> > - we can also roll our own mix alias using https://github.com/okeuday/reltool_util
> > - latest elixir versions have `release` command out of the box
> > 8. Raw dependencies are supported via `compile: false` option
> > 9. I think it is doable. In the worst case scenario we could just tar the content of `src` directory after calling `mix deps.get`
> >
> > # Warts of mix in the context of CouchDB project
> >
> > - CouchDB doesn't use standard elixir directories layout which caused include_lib directives to not work. To solve this issue absolute path to `<root>/src` need to be added into include path
> > - for dependencies managed by mix it is done via `erlc_options` parameter
> > - Unfortunately when we call out to rebar there is no way to modify arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS` environment variable. Which is not very elegant solution.
>
> This may also cause problems on Windows, where we have very specific
> requirements for compiler options so as to ensure the correct libraries
> are linked for couchjs and so on.
>
> Have you tried compiling on that platform? ;) (Please don't make me do
> all the work here, again...)
>
> > - We need to create empty `rebar.config` file for erlang applications which do not have `rebar.config` (all applications in couchdb repo). This problem can be fixed by adding mix.exs to dependencies we have control over.
> > - The mix is slow when we have to call out to rebar. This problem can be fixed by adding mix.exs to dependencies we have control over.
>
> How much slower? Can you post a comparison of `time make couch` between
> master and your branch?
master:
> time bin/rebar get-deps update-deps
real 0m37.514s
user 0m1.960s
sys 0m3.300s
> time make couch
real 0m49.677s
user 0m46.910s
sys 0m22.210s
mix:
> time mix do deps.get
real 0m24.405s
user 0m1.590s
sys 0m1.270s
> time mix deps.compile
real 3m47.968s
user 0m0.860s
sys 0m0.640s
> > # Roadmap
> >
> > The full conversion would take some time and if we decide to go with mix it makes sense to do it incrementally. Possible roadmap could be:
> > - initial PR
> > - offload dependency fetching to mix.exs and generate rebar.config so release machinery still works
> > - Update Makefile targets to call mix
> > - Choose or implement eunit plugin for mix
> > - Support running tests individually (replacement for `make eunit apps= tests= suites=`)
> > - Replace rebar plugins we use with alternative solutions
> > - Figure out how to use dialyzer
> > - Figure out how to use mix for preparing release
>
> Anything that lands on master must support release preparation. If your
> incremental approach breaks `make dist`, you'll need to stage all of
> your incremental changes on a feature branch and fix `make dist` before
> merging to master. master needs to be releasable at all times.
+1000
> > Best regards,
> > ILYA (aka iilyak)
>
> -Joan "release engineer hat is now off" Touzet
>
>
Re: [PROPOSAL] Gradually replace rebar with mix
Posted by Joan Touzet <wo...@apache.org>.
Hi Ilya,
On 2019-08-15 10:09, Ilya Khlopotov wrote:
> Hello,
>
> There is an https://github.com/apache/couchdb/issues/1428 issue about migrating to rebar3 or mix. I did an experiment to switch from rebar into mix and wanted to share the results. The code for experiment is here https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. Overall I am happy with the experiment.
>
> # Requirements
>
> 1. Should be able to compile dependencies which use either rebar or rebar3
> 2. Should be able to lock dependencies to specific versions using references to commits
> 3. Should be able to call eunit test suite
> 4. Should be able to compile NIFs
> 5. Should be easy to install plugins
> 6. Should support dialyzer
> 7. Should support release managers
> 8. Support for raw dependencies
> 9. Produce source tarball suitable for offline compilation (ASF requirement)
>
> # How mix supports the requirements
>
> 1. Mix is able to figure out which build tool to use. There is also a possibility to specify build tool individually for a given dependency.
> `manager: mix | rebar | rebar3 | make`. There is also support for calling arbitrary shell command configured via `compile: <command>`. The manual step is required to install the needed tools `mix local.rebar --force`.
> However there are means to automate that step (we can use Makefile or define a `bootstrap` mix alias )
> 2. Dependencies are locked via `mix.lock` file.
I don't have any strong opinions one way or the other, but I did see
Garren asking about 'hex' instead of / in conjunction with 'mix' for
dependency management. Is this effort combined with his or independent?
I don't know the tools well enough.
> 3. Eunit is supported via one of the mix plugins. Both plugins are very old but it should be easy to write our own (32 lines of code).
> - https://github.com/dantswain/mix_eunit
> - https://github.com/talentdeficit/mixunit
> 4. NIFs can be compiled using one of the following approaches:
> - specify `rebar` as build manager for a dependency (easiest)
Are you considering staying with rebar2 here, or moving to rebar(3)? It
would be nice not to have to keep supporting (and shipping!) our fork of
rebar2.
> - call a Makefile from mix.exs. See examples:
> - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
> - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
> - use https://github.com/davisp/erlang-native-compiler
> - use https://github.com/blt/port_compiler
Paul left a comment about this on the roadmap issue just today. Can you
respond to his comment (it was related tot dependencies-of-dependencies,
i.e. jiffy)
He also raised the concern that it may force us to use only the very
latest Erlang versions - can you comment on this? Does the current
hex/mix situation require a minimum version? What version is that?
> 5. Installing plugins for mix is similar to rebar3. It is sufficient to just specify list of plugins in the deps in mix.exs
> 6. There are multiple dyalizer plugins to choose from
> - https://github.com/jeremyjh/dialyxir
> - https://github.com/gabrielgatu/mix-dialyzer
I know this is a requirement for Cloudant, but today isn't for base
CouchDB. I would prefer not to introduce this requirement at the same
time as migrating build tool.
> 7. Multiple release managers are supported via mix plugins
> - distillery is supported via https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
> - relx and reltools can be used via https://github.com/bitwalker/exrm (deprecated in favor of distillery)
> - we can also roll our own mix alias using https://github.com/okeuday/reltool_util
> - latest elixir versions have `release` command out of the box
> 8. Raw dependencies are supported via `compile: false` option
> 9. I think it is doable. In the worst case scenario we could just tar the content of `src` directory after calling `mix deps.get`
>
> # Warts of mix in the context of CouchDB project
>
> - CouchDB doesn't use standard elixir directories layout which caused include_lib directives to not work. To solve this issue absolute path to `<root>/src` need to be added into include path
> - for dependencies managed by mix it is done via `erlc_options` parameter
> - Unfortunately when we call out to rebar there is no way to modify arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS` environment variable. Which is not very elegant solution.
This may also cause problems on Windows, where we have very specific
requirements for compiler options so as to ensure the correct libraries
are linked for couchjs and so on.
Have you tried compiling on that platform? ;) (Please don't make me do
all the work here, again...)
> - We need to create empty `rebar.config` file for erlang applications which do not have `rebar.config` (all applications in couchdb repo). This problem can be fixed by adding mix.exs to dependencies we have control over.
> - The mix is slow when we have to call out to rebar. This problem can be fixed by adding mix.exs to dependencies we have control over.
How much slower? Can you post a comparison of `time make couch` between
master and your branch?
> # Roadmap
>
> The full conversion would take some time and if we decide to go with mix it makes sense to do it incrementally. Possible roadmap could be:
> - initial PR
> - offload dependency fetching to mix.exs and generate rebar.config so release machinery still works
> - Update Makefile targets to call mix
> - Choose or implement eunit plugin for mix
> - Support running tests individually (replacement for `make eunit apps= tests= suites=`)
> - Replace rebar plugins we use with alternative solutions
> - Figure out how to use dialyzer
> - Figure out how to use mix for preparing release
Anything that lands on master must support release preparation. If your
incremental approach breaks `make dist`, you'll need to stage all of
your incremental changes on a feature branch and fix `make dist` before
merging to master. master needs to be releasable at all times.
> Best regards,
> ILYA (aka iilyak)
-Joan "release engineer hat is now off" Touzet