You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2019/07/07 13:47:39 UTC
Request to test the new CI test environment / CI Docker Image
Hello everyone,
I think after the last round of reviews from Ash, I will be quite ready
this week to merge the final 3/3 step of the new CI Docker image.
Since it impacts integration-testing workflow of all Airflow developers, I
have a kind request to people who develop Airflow to test it before we
merge it to master.
The CI builds were extensively tested over the past few months so I am
quite confident it will be ok for Travis, but there are quite a few
improvements for local development.
Among others you will be able to:
- Pull the CI images from DockerHub and build your local version that
you will be use for testing via "local_ci_build.sh" (and force pull with
another script)
- Run "local_ci_run_airflow_testing.sh" <test_name> to run a test
- Run "local_ci_enter_environment.sh" to enter the local IT environment
being 1-1 to what you have in Travis CI and run tests more quickly (also
with --skip-db-init flag for even faster runs)
- Run static code analysis (mypy/pylint/flake8/doc build/licence check)
in 1-1 the same way as you run them in Travis using "
ci_pylint.sh/ci_mypy.sh ...."
I prepared the documentation update in CONTRIBUTING about "Integration test
development environment"
https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#integration-test-development-environment
You can check the branch out at
https://github.com/PolideaInternal/airflow/tree/ms-travis-ci-tests
There are more improvements to come/propose soon - pre-commits, possibly
full "simplified development environment" (which follows next as AIP-7),
production-optimised official Airflow image, but it is already quite a
change (improvement I believe) in the development workflow for Airflow.
I would love to hear if AIP-10 works as intended for you as well as for the
few people who used it/tested so far. Please let me know your
comments/questions - I am happy to answer them. Especially if you have
Linux desktop - I would love to get it tested.
If I have enough comments till Monday evening, I would love to merge it on
Tuesday.
J
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Request to test the new CI test environment / CI Docker Image
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hey Stefan (also anyone else who wants to make last quick check):
In the latest iteration I think I solved most Linux problems (and added a
few more things) - you can check it yourself:
https://github.com/PolideaInternal/airflow/tree/ms-travis-ci-tests
- I got rid of .pyc files. The .pyc files and __pychache_ folders will
be deleted on entering the container and no new .pyc files will be created
(negligible import time penalty, no impact on run performance)
- Ownership of the files is fixed automatically in case we build
documentation (scripts/ci/ci_docs.sh)
- You can also fix ownership for any files manually by
(scripts/ci/local_ci_fix_ownership.sh)
- You can run all static checks from within the container not only from
the host script
- You can run all static checks selectively - for the whole package or
single python file.
- The last static check - check_licence is also dockerised now - no more
java/downloaded rat.jar needed locally to run licence check
All docs are cleaned up and updated in
https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#integration-test-development-environment
J.
On Wed, Jul 10, 2019 at 8:50 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Yeah. It's the optimisation I mentioned that's missing now.
>
> But I realised that you can (and it's easy) run the ci/mypy/flake once you
> enter the environment. I will add a few scripts so that you can just run
> scripts (and will update the docs).
>
> The scripts to run on host are mainly for CI purpose now but in the near
> future I will use them to automatically run tests during the pre-commit
> hooks (and there I have optimisation that the "rebuild" happen only when
> needed. And there I have implemented switching to python3 automatically. I
> might actually port most of it even now I think.
>
> Thanks for the feedbac. Very valuable!
>
> J.
>
> On Wed, Jul 10, 2019 at 8:18 AM Stefan Seelmann <ma...@stefan-seelmann.de>
> wrote:
>
>> On 7/9/19 7:36 PM, Jarek Potiuk wrote:
>> > - The warning with /root/.docker/config.json is nothing to worry
>> about
>> > (it tries to access your host configuration for credentials - but we
>> do not
>> > need any credentials for the environment).
>>
>> Ok, fine
>>
>> > - The CLEAN_FILES unbound variable was an interesting one - not sure
>> why
>> > it was not set at the first run and set it on the other (likely
>> because
>> > CLEAN_FILES is propagated through docker-compose settings and set to
>> empty
>> > after initialisation) but I protected against such case now
>>
>> Works now.
>>
>> > - Running flake/pylint is not supposed to be run from the container
>> but
>> > from the host. I looked at the docs and realised it was not at all
>> clear
>> > :). It was obvious for me but not for someone new :). That's big
>> value of
>> > such tests. I updated CONTRIBUTING.md with a note and also added
>> extra
>> > protection.
>>
>> When I just run `./scripts/ci/ci_flake8.sh` it attempts to build the
>> Python 2.7 image `apache/airflow:master-python2.7-ci-slim` (I assume
>> because my default Python interpreter is 2.7) which fails.
>>
>> When I run `PYTHON_VERSION=3.6 ./scripts/ci/ci_flake8.sh` it works.
>> However on every invocation it rebuilds the "Airflow CI slim image", it
>> uses cached layers, but still takes 23 seconds, it that wanted?
>>
>> Kind Regards,
>> Stefan
>>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Request to test the new CI test environment / CI Docker Image
Posted by Jarek Potiuk <Ja...@polidea.com>.
Yeah. It's the optimisation I mentioned that's missing now.
But I realised that you can (and it's easy) run the ci/mypy/flake once you
enter the environment. I will add a few scripts so that you can just run
scripts (and will update the docs).
The scripts to run on host are mainly for CI purpose now but in the near
future I will use them to automatically run tests during the pre-commit
hooks (and there I have optimisation that the "rebuild" happen only when
needed. And there I have implemented switching to python3 automatically. I
might actually port most of it even now I think.
Thanks for the feedbac. Very valuable!
J.
On Wed, Jul 10, 2019 at 8:18 AM Stefan Seelmann <ma...@stefan-seelmann.de>
wrote:
> On 7/9/19 7:36 PM, Jarek Potiuk wrote:
> > - The warning with /root/.docker/config.json is nothing to worry about
> > (it tries to access your host configuration for credentials - but we
> do not
> > need any credentials for the environment).
>
> Ok, fine
>
> > - The CLEAN_FILES unbound variable was an interesting one - not sure
> why
> > it was not set at the first run and set it on the other (likely
> because
> > CLEAN_FILES is propagated through docker-compose settings and set to
> empty
> > after initialisation) but I protected against such case now
>
> Works now.
>
> > - Running flake/pylint is not supposed to be run from the container
> but
> > from the host. I looked at the docs and realised it was not at all
> clear
> > :). It was obvious for me but not for someone new :). That's big
> value of
> > such tests. I updated CONTRIBUTING.md with a note and also added extra
> > protection.
>
> When I just run `./scripts/ci/ci_flake8.sh` it attempts to build the
> Python 2.7 image `apache/airflow:master-python2.7-ci-slim` (I assume
> because my default Python interpreter is 2.7) which fails.
>
> When I run `PYTHON_VERSION=3.6 ./scripts/ci/ci_flake8.sh` it works.
> However on every invocation it rebuilds the "Airflow CI slim image", it
> uses cached layers, but still takes 23 seconds, it that wanted?
>
> Kind Regards,
> Stefan
>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Request to test the new CI test environment / CI Docker Image
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 7/9/19 7:36 PM, Jarek Potiuk wrote:
> - The warning with /root/.docker/config.json is nothing to worry about
> (it tries to access your host configuration for credentials - but we do not
> need any credentials for the environment).
Ok, fine
> - The CLEAN_FILES unbound variable was an interesting one - not sure why
> it was not set at the first run and set it on the other (likely because
> CLEAN_FILES is propagated through docker-compose settings and set to empty
> after initialisation) but I protected against such case now
Works now.
> - Running flake/pylint is not supposed to be run from the container but
> from the host. I looked at the docs and realised it was not at all clear
> :). It was obvious for me but not for someone new :). That's big value of
> such tests. I updated CONTRIBUTING.md with a note and also added extra
> protection.
When I just run `./scripts/ci/ci_flake8.sh` it attempts to build the
Python 2.7 image `apache/airflow:master-python2.7-ci-slim` (I assume
because my default Python interpreter is 2.7) which fails.
When I run `PYTHON_VERSION=3.6 ./scripts/ci/ci_flake8.sh` it works.
However on every invocation it rebuilds the "Airflow CI slim image", it
uses cached layers, but still takes 23 seconds, it that wanted?
Kind Regards,
Stefan
Re: Request to test the new CI test environment / CI Docker Image
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hey Stefan,
All the important problems should be fixed now! You can pull the latest
changes from my branch and retry:
- The warning with /root/.docker/config.json is nothing to worry about
(it tries to access your host configuration for credentials - but we do not
need any credentials for the environment).
- The CLEAN_FILES unbound variable was an interesting one - not sure why
it was not set at the first run and set it on the other (likely because
CLEAN_FILES is propagated through docker-compose settings and set to empty
after initialisation) but I protected against such case now
- Running flake/pylint is not supposed to be run from the container but
from the host. I looked at the docs and realised it was not at all clear
:). It was obvious for me but not for someone new :). That's big value of
such tests. I updated CONTRIBUTING.md with a note and also added extra
protection. When you try those scripts from container you will get the
following error now:
*You are inside the Airflow docker container!You should only run this
script from the host.Learn more about how we develop and test airflow in:*
*https://github.com/apache/airflow/blob/master/CONTRIBUTING.md
<https://github.com/apache/airflow/blob/master/CONTRIBUTING.md> *
Please try to run them from the host (and after pull you can try from the
container and you will see the error). It will run a bit slower now because
it checks whether docker should be rebuilt but soon I will optimise it away
with pre-commit hooks (I already have a POC working).
Anyone else wants to try - feel free and let me know!
J.
On Tue, Jul 9, 2019 at 7:03 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Thanks Stefan! - will look into all those and fix :). Those are some of
> the small things that should be super easy to fix.
>
> The root user for Linux is is one of the differences Mac <> Linux that is
> super difficult to resolve (in current Docker) when you mount local
> volumes. The problem is that inside the container you will have the same
> uid/gid as your host user but you will not have that user inside the
> container (and then you will have no permissions to run airflow inside the
> container). Unfortunately mapping the users host <-> container is not easy
> (requires engine restart and cannot be done per-container). And we cannot
> assume fixed UID (different host users have different ids).
>
> Here is detailed explanation from our PR:
> https://github.com/apache/airflow/pull/4938#issuecomment-507790924
>
> This problem does not occur on MacOS because on Mac there is a (slower)
> osxfs filesystem used that maps users from host to linux VM -> and then to
> container. I am going to workaround it very soon for Linux as well.. In the
> follow-up "simplified development environment" AIP-7 those root-owned files
> will be cleaned up on entering the environment and you will have a simple
> command to clean them up. Maybe not a perfect solution but it is better
> than manual cleanup with sudo.
>
> J.
>
>
> On Tue, Jul 9, 2019 at 6:38 PM Stefan Seelmann <ma...@stefan-seelmann.de>
> wrote:
>
>> Hi Jarek,
>>
>> tested quickly on ArchLinux. Overall looks very good, thanks for all the
>> effort!
>>
>>
>> When building the images for the first time I saw the following warning:
>> ```
>> WARNING: Error loading config file: /root/.docker/config.json: stat
>> /root/.docker/config.json: permission denied
>> ```
>>
>>
>> When running `./scripts/ci/local_ci_enter_environment.sh` the first time
>> it fails:
>> ```
>> $ PYTHON_VERSION=3.6 BACKEND=postgres ENV=docker
>> ./scripts/ci/local_ci_enter_environment.sh
>> /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh: line 57:
>> CLEAN_FILES: unbound variable
>> ```
>> I added `CLEAN_FILES=false` to the command then it worked. Next time the
>> additional env variable was not required.
>>
>>
>> Within the dev env I was able to run test using `run-tests`.
>>
>>
>> However running flake8 or pylint fails, seems the `hooks` folder is not
>> mounted into the docker container:
>> ```
>> /opt/airflow /opt/airflow
>> scripts/ci/ci_flake8.sh: line 32: ./hooks/build: No such file or directory
>> ```
>>
>>
>> Another observation: Files and folders created by the environment have
>> user/group `root` (examples are `logs` and `__pycache__`). I assume the
>> reason is that the container runs as user root. Maybe it's possible to
>> run the container with uid/gid of the curren user using `-u $(id
>> -u):$(id -g)`?
>>
>>
>> Kind Regards,
>> Stefan
>>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Request to test the new CI test environment / CI Docker Image
Posted by Jarek Potiuk <Ja...@polidea.com>.
Thanks Stefan! - will look into all those and fix :). Those are some of the
small things that should be super easy to fix.
The root user for Linux is is one of the differences Mac <> Linux that is
super difficult to resolve (in current Docker) when you mount local
volumes. The problem is that inside the container you will have the same
uid/gid as your host user but you will not have that user inside the
container (and then you will have no permissions to run airflow inside the
container). Unfortunately mapping the users host <-> container is not easy
(requires engine restart and cannot be done per-container). And we cannot
assume fixed UID (different host users have different ids).
Here is detailed explanation from our PR:
https://github.com/apache/airflow/pull/4938#issuecomment-507790924
This problem does not occur on MacOS because on Mac there is a (slower)
osxfs filesystem used that maps users from host to linux VM -> and then to
container. I am going to workaround it very soon for Linux as well.. In the
follow-up "simplified development environment" AIP-7 those root-owned files
will be cleaned up on entering the environment and you will have a simple
command to clean them up. Maybe not a perfect solution but it is better
than manual cleanup with sudo.
J.
On Tue, Jul 9, 2019 at 6:38 PM Stefan Seelmann <ma...@stefan-seelmann.de>
wrote:
> Hi Jarek,
>
> tested quickly on ArchLinux. Overall looks very good, thanks for all the
> effort!
>
>
> When building the images for the first time I saw the following warning:
> ```
> WARNING: Error loading config file: /root/.docker/config.json: stat
> /root/.docker/config.json: permission denied
> ```
>
>
> When running `./scripts/ci/local_ci_enter_environment.sh` the first time
> it fails:
> ```
> $ PYTHON_VERSION=3.6 BACKEND=postgres ENV=docker
> ./scripts/ci/local_ci_enter_environment.sh
> /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh: line 57:
> CLEAN_FILES: unbound variable
> ```
> I added `CLEAN_FILES=false` to the command then it worked. Next time the
> additional env variable was not required.
>
>
> Within the dev env I was able to run test using `run-tests`.
>
>
> However running flake8 or pylint fails, seems the `hooks` folder is not
> mounted into the docker container:
> ```
> /opt/airflow /opt/airflow
> scripts/ci/ci_flake8.sh: line 32: ./hooks/build: No such file or directory
> ```
>
>
> Another observation: Files and folders created by the environment have
> user/group `root` (examples are `logs` and `__pycache__`). I assume the
> reason is that the container runs as user root. Maybe it's possible to
> run the container with uid/gid of the curren user using `-u $(id
> -u):$(id -g)`?
>
>
> Kind Regards,
> Stefan
>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Request to test the new CI test environment / CI Docker Image
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
Hi Jarek,
tested quickly on ArchLinux. Overall looks very good, thanks for all the
effort!
When building the images for the first time I saw the following warning:
```
WARNING: Error loading config file: /root/.docker/config.json: stat
/root/.docker/config.json: permission denied
```
When running `./scripts/ci/local_ci_enter_environment.sh` the first time
it fails:
```
$ PYTHON_VERSION=3.6 BACKEND=postgres ENV=docker
./scripts/ci/local_ci_enter_environment.sh
/opt/airflow/scripts/ci/in_container/entrypoint_ci.sh: line 57:
CLEAN_FILES: unbound variable
```
I added `CLEAN_FILES=false` to the command then it worked. Next time the
additional env variable was not required.
Within the dev env I was able to run test using `run-tests`.
However running flake8 or pylint fails, seems the `hooks` folder is not
mounted into the docker container:
```
/opt/airflow /opt/airflow
scripts/ci/ci_flake8.sh: line 32: ./hooks/build: No such file or directory
```
Another observation: Files and folders created by the environment have
user/group `root` (examples are `logs` and `__pycache__`). I assume the
reason is that the container runs as user root. Maybe it's possible to
run the container with uid/gid of the curren user using `-u $(id
-u):$(id -g)`?
Kind Regards,
Stefan
Re: Request to test the new CI test environment / CI Docker Image
Posted by Jarek Potiuk <Ja...@polidea.com>.
After testing with Daniell I added fixes for a few "teething" problems. I
think the version we have now is pretty solid. If there is anyone willing
to test it still (ideally on Linux), that would be great.
Docs are here:
https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#integration-test-development-environment
You can check-out the branch at:
https://github.com/PolideaInternal/airflow/tree/ms-travis-ci-tests
Thanks Daniel again for your help! I hope we can get it merged really soon.
J.
On Mon, Jul 8, 2019 at 8:22 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> OK. The problem is likely solved. It was due to having host-installed
> `apache_airflow.egg-info` with Werkzeug dependency still there mounted to
> docker container.
>
> The latest commit in my branch solves it - only mounting what's really
> needed. So it should be no problem any more (at least for the new
> environment). Separate mail with Werkzeug for local virtualenvs follows.
>
> J.
>
> On Mon, Jul 8, 2019 at 7:56 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> First feedback (thanks to Daniel) - due to the werkzeug dependency
>> recent removal - there might be a problem with running the test environment
>> - I am looking into it - but if you try and have "Werkzeug dependency"
>> problem when you try it - I am working on it. But if you try and you do not
>> have the problem - also let me know - that might be helpful because I am
>> looking for the reason why it works for me and does not work for Daniel.
>>
>> Thanks Daniel for helping with it !
>>
>> J
>>
>> On Sun, Jul 7, 2019 at 9:21 PM Daniel Imberman <da...@gmail.com>
>> wrote:
>>
>>> Hi Jarek,
>>>
>>> I will look into this today and tomorrow. Will report back any
>>> feedback/findings :).
>>>
>>> Daniel
>>>
>>> On Sun, Jul 7, 2019 at 6:47 AM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> I think after the last round of reviews from Ash, I will be quite ready
>>>> this week to merge the final 3/3 step of the new CI Docker image.
>>>>
>>>> Since it impacts integration-testing workflow of all Airflow
>>>> developers, I have a kind request to people who develop Airflow to test it
>>>> before we merge it to master.
>>>>
>>>> The CI builds were extensively tested over the past few months so I am
>>>> quite confident it will be ok for Travis, but there are quite a few
>>>> improvements for local development.
>>>>
>>>> Among others you will be able to:
>>>>
>>>> - Pull the CI images from DockerHub and build your local version
>>>> that you will be use for testing via "local_ci_build.sh" (and force pull
>>>> with another script)
>>>> - Run "local_ci_run_airflow_testing.sh" <test_name> to run a test
>>>> - Run "local_ci_enter_environment.sh" to enter the local IT
>>>> environment being 1-1 to what you have in Travis CI and run tests more
>>>> quickly (also with --skip-db-init flag for even faster runs)
>>>> - Run static code analysis (mypy/pylint/flake8/doc build/licence
>>>> check) in 1-1 the same way as you run them in Travis using "
>>>> ci_pylint.sh/ci_mypy.sh ...."
>>>>
>>>> I prepared the documentation update in CONTRIBUTING about "Integration
>>>> test development environment"
>>>> https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#integration-test-development-environment
>>>>
>>>> You can check the branch out at
>>>> https://github.com/PolideaInternal/airflow/tree/ms-travis-ci-tests
>>>>
>>>> There are more improvements to come/propose soon - pre-commits,
>>>> possibly full "simplified development environment" (which follows next as
>>>> AIP-7), production-optimised official Airflow image, but it is already
>>>> quite a change (improvement I believe) in the development workflow for
>>>> Airflow.
>>>>
>>>> I would love to hear if AIP-10 works as intended for you as well as for
>>>> the few people who used it/tested so far. Please let me know your
>>>> comments/questions - I am happy to answer them. Especially if you have
>>>> Linux desktop - I would love to get it tested.
>>>>
>>>> If I have enough comments till Monday evening, I would love to merge it
>>>> on Tuesday.
>>>>
>>>> J
>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Request to test the new CI test environment / CI Docker Image
Posted by Jarek Potiuk <Ja...@polidea.com>.
OK. The problem is likely solved. It was due to having host-installed
`apache_airflow.egg-info` with Werkzeug dependency still there mounted to
docker container.
The latest commit in my branch solves it - only mounting what's really
needed. So it should be no problem any more (at least for the new
environment). Separate mail with Werkzeug for local virtualenvs follows.
J.
On Mon, Jul 8, 2019 at 7:56 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> First feedback (thanks to Daniel) - due to the werkzeug dependency recent
> removal - there might be a problem with running the test environment - I am
> looking into it - but if you try and have "Werkzeug dependency" problem
> when you try it - I am working on it. But if you try and you do not have
> the problem - also let me know - that might be helpful because I am looking
> for the reason why it works for me and does not work for Daniel.
>
> Thanks Daniel for helping with it !
>
> J
>
> On Sun, Jul 7, 2019 at 9:21 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> Hi Jarek,
>>
>> I will look into this today and tomorrow. Will report back any
>> feedback/findings :).
>>
>> Daniel
>>
>> On Sun, Jul 7, 2019 at 6:47 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> I think after the last round of reviews from Ash, I will be quite ready
>>> this week to merge the final 3/3 step of the new CI Docker image.
>>>
>>> Since it impacts integration-testing workflow of all Airflow developers,
>>> I have a kind request to people who develop Airflow to test it before we
>>> merge it to master.
>>>
>>> The CI builds were extensively tested over the past few months so I am
>>> quite confident it will be ok for Travis, but there are quite a few
>>> improvements for local development.
>>>
>>> Among others you will be able to:
>>>
>>> - Pull the CI images from DockerHub and build your local version
>>> that you will be use for testing via "local_ci_build.sh" (and force pull
>>> with another script)
>>> - Run "local_ci_run_airflow_testing.sh" <test_name> to run a test
>>> - Run "local_ci_enter_environment.sh" to enter the local IT
>>> environment being 1-1 to what you have in Travis CI and run tests more
>>> quickly (also with --skip-db-init flag for even faster runs)
>>> - Run static code analysis (mypy/pylint/flake8/doc build/licence
>>> check) in 1-1 the same way as you run them in Travis using "
>>> ci_pylint.sh/ci_mypy.sh ...."
>>>
>>> I prepared the documentation update in CONTRIBUTING about "Integration
>>> test development environment"
>>> https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#integration-test-development-environment
>>>
>>> You can check the branch out at
>>> https://github.com/PolideaInternal/airflow/tree/ms-travis-ci-tests
>>>
>>> There are more improvements to come/propose soon - pre-commits, possibly
>>> full "simplified development environment" (which follows next as AIP-7),
>>> production-optimised official Airflow image, but it is already quite a
>>> change (improvement I believe) in the development workflow for Airflow.
>>>
>>> I would love to hear if AIP-10 works as intended for you as well as for
>>> the few people who used it/tested so far. Please let me know your
>>> comments/questions - I am happy to answer them. Especially if you have
>>> Linux desktop - I would love to get it tested.
>>>
>>> If I have enough comments till Monday evening, I would love to merge it
>>> on Tuesday.
>>>
>>> J
>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Request to test the new CI test environment / CI Docker Image
Posted by Jarek Potiuk <Ja...@polidea.com>.
First feedback (thanks to Daniel) - due to the werkzeug dependency recent
removal - there might be a problem with running the test environment - I am
looking into it - but if you try and have "Werkzeug dependency" problem
when you try it - I am working on it. But if you try and you do not have
the problem - also let me know - that might be helpful because I am looking
for the reason why it works for me and does not work for Daniel.
Thanks Daniel for helping with it !
J
On Sun, Jul 7, 2019 at 9:21 PM Daniel Imberman <da...@gmail.com>
wrote:
> Hi Jarek,
>
> I will look into this today and tomorrow. Will report back any
> feedback/findings :).
>
> Daniel
>
> On Sun, Jul 7, 2019 at 6:47 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Hello everyone,
>>
>> I think after the last round of reviews from Ash, I will be quite ready
>> this week to merge the final 3/3 step of the new CI Docker image.
>>
>> Since it impacts integration-testing workflow of all Airflow developers,
>> I have a kind request to people who develop Airflow to test it before we
>> merge it to master.
>>
>> The CI builds were extensively tested over the past few months so I am
>> quite confident it will be ok for Travis, but there are quite a few
>> improvements for local development.
>>
>> Among others you will be able to:
>>
>> - Pull the CI images from DockerHub and build your local version that
>> you will be use for testing via "local_ci_build.sh" (and force pull with
>> another script)
>> - Run "local_ci_run_airflow_testing.sh" <test_name> to run a test
>> - Run "local_ci_enter_environment.sh" to enter the local IT
>> environment being 1-1 to what you have in Travis CI and run tests more
>> quickly (also with --skip-db-init flag for even faster runs)
>> - Run static code analysis (mypy/pylint/flake8/doc build/licence
>> check) in 1-1 the same way as you run them in Travis using "
>> ci_pylint.sh/ci_mypy.sh ...."
>>
>> I prepared the documentation update in CONTRIBUTING about "Integration
>> test development environment"
>> https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#integration-test-development-environment
>>
>> You can check the branch out at
>> https://github.com/PolideaInternal/airflow/tree/ms-travis-ci-tests
>>
>> There are more improvements to come/propose soon - pre-commits, possibly
>> full "simplified development environment" (which follows next as AIP-7),
>> production-optimised official Airflow image, but it is already quite a
>> change (improvement I believe) in the development workflow for Airflow.
>>
>> I would love to hear if AIP-10 works as intended for you as well as for
>> the few people who used it/tested so far. Please let me know your
>> comments/questions - I am happy to answer them. Especially if you have
>> Linux desktop - I would love to get it tested.
>>
>> If I have enough comments till Monday evening, I would love to merge it
>> on Tuesday.
>>
>> J
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Request to test the new CI test environment / CI Docker Image
Posted by Daniel Imberman <da...@gmail.com>.
Hi Jarek,
I will look into this today and tomorrow. Will report back any
feedback/findings :).
Daniel
On Sun, Jul 7, 2019 at 6:47 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Hello everyone,
>
> I think after the last round of reviews from Ash, I will be quite ready
> this week to merge the final 3/3 step of the new CI Docker image.
>
> Since it impacts integration-testing workflow of all Airflow developers, I
> have a kind request to people who develop Airflow to test it before we
> merge it to master.
>
> The CI builds were extensively tested over the past few months so I am
> quite confident it will be ok for Travis, but there are quite a few
> improvements for local development.
>
> Among others you will be able to:
>
> - Pull the CI images from DockerHub and build your local version that
> you will be use for testing via "local_ci_build.sh" (and force pull with
> another script)
> - Run "local_ci_run_airflow_testing.sh" <test_name> to run a test
> - Run "local_ci_enter_environment.sh" to enter the local IT
> environment being 1-1 to what you have in Travis CI and run tests more
> quickly (also with --skip-db-init flag for even faster runs)
> - Run static code analysis (mypy/pylint/flake8/doc build/licence
> check) in 1-1 the same way as you run them in Travis using "
> ci_pylint.sh/ci_mypy.sh ...."
>
> I prepared the documentation update in CONTRIBUTING about "Integration
> test development environment"
> https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#integration-test-development-environment
>
> You can check the branch out at
> https://github.com/PolideaInternal/airflow/tree/ms-travis-ci-tests
>
> There are more improvements to come/propose soon - pre-commits, possibly
> full "simplified development environment" (which follows next as AIP-7),
> production-optimised official Airflow image, but it is already quite a
> change (improvement I believe) in the development workflow for Airflow.
>
> I would love to hear if AIP-10 works as intended for you as well as for
> the few people who used it/tested so far. Please let me know your
> comments/questions - I am happy to answer them. Especially if you have
> Linux desktop - I would love to get it tested.
>
> If I have enough comments till Monday evening, I would love to merge it on
> Tuesday.
>
> J
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>