[jira] [Created] (AIRFLOW-2499) Dockerised CI pipeline

["Gerardo Curiel (JIRA)" <ji...@apache.org>]

Gerardo Curiel created AIRFLOW-2499:
---------------------------------------

             Summary: Dockerised CI pipeline
                 Key: AIRFLOW-2499
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2499
             Project: Apache Airflow
          Issue Type: Test
          Components: ci, tests
    Affects Versions: 1.10.0, 1.10
            Reporter: Gerardo Curiel
            Assignee: Gerardo Curiel
             Fix For: 1.10.0, 1.10


Currently, running unit tests is a difficult process. Airflow tests depend on many external services and other custom setup, which makes it hard for contributors to work on this codebase. CI builds have also been unreliable[0][1][2][3], and it is hard to reproduce the causes. Having contributors trying to emulate the build environment every time makes it easier to get to an "it works on my machine" sort of situation.

This PR implements a dockerised version of the current build pipeline. This setup has a few advantages:
 * TravisCI tests are reproducible locally
 * The same build setup can be used to create a local development environment (there's a request for it [4])

 

*Implementation details*
 * I'm using Docker Compose for the container orchestration and configuration.
 * MySQL, PostgreSQL, OpenLDAP, krb5 and rabbitmq are now services running inside their own containers
 * I created a separate repo, called incubator-airflow-ci[5] (TravisCI build here[6]), where a base image with all dependencies is built. In this case, I'm following the same pattern the CouchDB[7] project follows
 * Hadoop, Hive and MiniCluster were moved to this base image
 * The current TravisCI pipeline lives here[8]. A few tests are still failing. It's still WIP.

 

*References*

[0] https://issues.apache.org/jira/browse/AIRFLOW-671
[1] https://issues.apache.org/jira/browse/AIRFLOW-968
[2] https://issues.apache.org/jira/browse/AIRFLOW-2157
[3] https://issues.apache.org/jira/browse/AIRFLOW-2272
[4] https://issues.apache.org/jira/browse/AIRFLOW-1042 
[5] https://github.com/gerardo/incubator-airflow-ci
[6] https://travis-ci.org/gerardo/incubator-airflow-ci
[7] https://travis-ci.org/apache/couchdb-ci
[8] https://travis-ci.org/gerardo/incubator-airflow

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)