[DISCUSS] Resolving dependencies with the new architecture

[Nadav Har Tzvi <na...@gmail.com>]

Hey everyone,

With Amaterasu-8 in place and Amaterasu-45 nearing completion, there is one
more critical thing we need to attend to before we could call the new
architecture complete.

Dependencies resolution, management, distribution.

A feedback I received in PyCon-IL last year was about not using the
standard Python dependencies resolution mechanisms. So for example, we
currently require to supply a "python.yml" file instead of a
"requirements.txt" file.
This could also be applied to the Scala counterpart. Instead of defining
the dependencies in the "jars.yml", we could leverage one or more of the
available build tools to manage this (maybe by supplying a pom.xml as part
of the job repository, somehow, or a build.gradle file)

In regard to Python specifically, going forward, we will definitely let the
user provide a requirements.txt file.

Dependencies distribution is also a problem I would like to think about.
Option 1:
Leader will merge job level and action level dependencies into one
dependencies definition file (e.g. requirements.txt) and will distribute
that file to the executors.
Pros:
Easy to implement.

Cons:
Will not work in networks without outgoing internet connection as the
executors will need access to an external package repository.

Option 2:
Leader will do a full dependencies resolution and distributed the already
downloaded and packaged dependencies to the executors, where they will be
installed.

Pros:
Will work in any environment, only requires the leader to have outgoing
internet connection.

Cons:
1. Much more difficult to implement. Requires implementation of wrappers to
language specific package management systems.
2. Introduces more state to the system that has to be managed. Another
"weak link" in my opinion.

Anyhow, let's discuss this, I would like to hear suggestions, thoughts, etc.

Cheers,
Nadav