[jira] [Resolved] (AIRFLOW-2499) Dockerised CI pipeline

["Fokko Driesprong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fokko Driesprong resolved AIRFLOW-2499.
---------------------------------------
    Resolution: Fixed

> Dockerised CI pipeline
> ----------------------
>
>                 Key: AIRFLOW-2499
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2499
>             Project: Apache Airflow
>          Issue Type: Test
>          Components: ci, tests
>    Affects Versions: 1.10.0, 1.10
>            Reporter: Gerardo Curiel
>            Assignee: Gerardo Curiel
>            Priority: Major
>              Labels: ci, docker, travis-ci
>             Fix For: 2.0.0
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> PR: https://github.com/apache/incubator-airflow/pull/3393
> Currently, running unit tests is a difficult process. Airflow tests depend on many external services and other custom setup, which makes it hard for contributors to work on this codebase. CI builds have also been unreliable[0][1][2][3], and it is hard to reproduce the causes. Having contributors trying to emulate the build environment every time makes it easier to get to an "it works on my machine" sort of situation.
> This PR implements a dockerised version of the current build pipeline. This setup has a few advantages:
>  * TravisCI tests are reproducible locally
>  * The same build setup can be used to create a local development environment (there's a request for it [4])
>  
> *Implementation details*
>  * I'm using Docker Compose for the container orchestration and configuration.
>  * MySQL, PostgreSQL, OpenLDAP, krb5 and rabbitmq are now services running inside their own containers
>  * I created a separate repo, called incubator-airflow-ci[5] (TravisCI build here[6]), where a base image with all dependencies is built. In this case, I'm following the same pattern the CouchDB[7] project follows
>  * Hadoop, Hive and MiniCluster were moved to this base image
>  * The current TravisCI pipeline lives here[8]. A few tests are still failing. It's still WIP.
>  
> *References*
> [0] https://issues.apache.org/jira/browse/AIRFLOW-671
>  [1] https://issues.apache.org/jira/browse/AIRFLOW-968
>  [2] https://issues.apache.org/jira/browse/AIRFLOW-2157
>  [3] https://issues.apache.org/jira/browse/AIRFLOW-2272
>  [4] https://issues.apache.org/jira/browse/AIRFLOW-1042 
>  [5] [https://github.com/gerardo/incubator-airflow-ci]
>  [6] [https://travis-ci.org/gerardo/incubator-airflow-ci]
>  [7] [https://travis-ci.org/apache/couchdb-ci]
>  [8] [https://travis-ci.org/gerardo/incubator-airflow]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)