[jira] [Updated] (AIRFLOW-5389) Better organised/faster build for external dependencies in ci build

["Kaxil Naik (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kaxil Naik updated AIRFLOW-5389:
--------------------------------
    Fix Version/s:     (was: 1.10.5)
                   1.10.6

> Better organised/faster build for external dependencies in ci build
> -------------------------------------------------------------------
>
>                 Key: AIRFLOW-5389
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5389
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: ci
>    Affects Versions: 2.0.0, 1.10.5
>            Reporter: Jarek Potiuk
>            Priority: Major
>             Fix For: 1.10.6
>
>
> There are several entries with if's for CI image in the Dockerfile. They have weird syntax and they are slow (each if line is separately evaluated while Docker build).
> They can be moved to bash scripts and replaced with single RUN command (one per CI deps and one for main file). Then we will also use the power of shellcheck to verify those scripts.
> In the future, we also think about unifying the dockerfiles later one (production/CI/SLIM without multi-staging) and removing installation of hive/hadoop/minicluster to separate images. This might be first step to do so.
> Examples of the current code:
> CI deps:
> {code:java}
> RUN \
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
>  mkdir -pv ${HADOOP_HOME} \
>  && mkdir -pv ${HIVE_HOME} \
>  && mkdir /tmp/minicluster \
>  && mkdir -pv /user/hive/warehouse \
>  && chmod -R 777 ${HIVE_HOME} \
>  && chmod -R 777 /user/ \
>  ;\
> fi
> # Install Hadoop
> # --absolute-names is a work around to avoid this issue https://github.com/docker/hub-feedback/issues/727
> RUN \
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
>     HADOOP_URL=${HADOOP_URL}hadoop-${HADOOP_VERSION}-${HADOOP_DISTRO}${HADOOP_DISTRO_VERSION}.tar.gz \
>     && HADOOP_TMP_FILE=/tmp/hadoop.tar.gz \
>     && curl -sL ${HADOOP_URL} > ${HADOOP_TMP_FILE} \
>     && tar xzf ${HADOOP_TMP_FILE} --absolute-names --strip-components 1 -C ${HADOOP_HOME} \
>     && rm ${HADOOP_TMP_FILE} \
>     ;\
> fi{code}
>  
> main:
> {code:java}
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
> chmod +x "${HOME}/run-tests-complete" "${HOME}/run-tests" && \
> chown "${AIRFLOW_USER}.${AIRFLOW_USER}" "${HOME}/.bashrc" \
>  "${HOME}/run-tests-complete" "${HOME}/run-tests"; \
> fi
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)