You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jarek Potiuk (Jira)" <ji...@apache.org> on 2019/09/02 20:38:00 UTC
[jira] [Created] (AIRFLOW-5389) Better organised/faster build for
external dependencies in ci build
Jarek Potiuk created AIRFLOW-5389:
-------------------------------------
Summary: Better organised/faster build for external dependencies in ci build
Key: AIRFLOW-5389
URL: https://issues.apache.org/jira/browse/AIRFLOW-5389
Project: Apache Airflow
Issue Type: Improvement
Components: ci
Affects Versions: 2.0.0, 1.10.5
Reporter: Jarek Potiuk
There are several entries with if's for CI image in the Dockerfile. They have weird syntax and they are slow (each if line is separately evaluated while Docker build).
They can be moved to bash scripts and replaced with single RUN command (one per CI deps and one for main file). Then we will also use the power of shellcheck to verify those scripts.
In the future, we also think about unifying the dockerfiles later one (production/CI/SLIM without multi-staging) and removing installation of hive/hadoop/minicluster to separate images. This might be first step to do so.
Examples of the current code:
CI deps:
{code:java}
RUN \
if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
mkdir -pv ${HADOOP_HOME} \
&& mkdir -pv ${HIVE_HOME} \
&& mkdir /tmp/minicluster \
&& mkdir -pv /user/hive/warehouse \
&& chmod -R 777 ${HIVE_HOME} \
&& chmod -R 777 /user/ \
;\
fi
# Install Hadoop
# --absolute-names is a work around to avoid this issue https://github.com/docker/hub-feedback/issues/727
RUN \
if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
HADOOP_URL=${HADOOP_URL}hadoop-${HADOOP_VERSION}-${HADOOP_DISTRO}${HADOOP_DISTRO_VERSION}.tar.gz \
&& HADOOP_TMP_FILE=/tmp/hadoop.tar.gz \
&& curl -sL ${HADOOP_URL} > ${HADOOP_TMP_FILE} \
&& tar xzf ${HADOOP_TMP_FILE} --absolute-names --strip-components 1 -C ${HADOOP_HOME} \
&& rm ${HADOOP_TMP_FILE} \
;\
fi{code}
main:
{code:java}
if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
chmod +x "${HOME}/run-tests-complete" "${HOME}/run-tests" && \
chown "${AIRFLOW_USER}.${AIRFLOW_USER}" "${HOME}/.bashrc" \
"${HOME}/run-tests-complete" "${HOME}/run-tests"; \
fi
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)