You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2021/01/21 20:19:39 UTC
[airflow] 14/44: Update CI to run tests againt v2-0-test branch
(#10891)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ba4858f9559b0a94a27445242e32396e622fc7a4
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Wed Dec 9 19:35:04 2020 +0000
Update CI to run tests againt v2-0-test branch (#10891)
(cherry picked from commit db166ba75c447a08b94e7be1ab09042fd6361581)
---
.github/workflows/ci.yml | 18 ++-
BREEZE.rst | 89 +++++++-----
CI.rst | 8 +-
breeze | 176 ++++++++++++++++++------
breeze-complete | 1 +
scripts/ci/constraints/ci_branch_constraints.sh | 2 +
6 files changed, 209 insertions(+), 85 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 44ffc7b..ccfd50f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -21,9 +21,9 @@ on: # yamllint disable-line rule:truthy
schedule:
- cron: '28 0 * * *'
push:
- branches: ['master', 'v1-10-test', 'v1-10-stable']
+ branches: ['master', 'v1-10-test', 'v1-10-stable', 'v2-0-test']
pull_request:
- branches: ['master', 'v1-10-test', 'v1-10-stable']
+ branches: ['master', 'v1-10-test', 'v1-10-stable', 'v2-0-test']
env:
@@ -745,7 +745,8 @@ jobs:
- verify-prod-images
- docs
if: >
- (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test') &&
+ (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ||
+ github.ref == 'refs/heads/v2-0-test') &&
github.event_name != 'schedule'
strategy:
matrix:
@@ -782,7 +783,8 @@ jobs:
- verify-ci-images
- docs
if: >
- (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ) &&
+ (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ||
+ github.ref == 'refs/heads/v2-0-test') &&
github.event_name != 'schedule'
strategy:
matrix:
@@ -817,7 +819,9 @@ jobs:
- ci-images
env:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test'
+ if: >
+ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ||
+ github.ref == 'refs/heads/v2-0-test'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -852,7 +856,9 @@ jobs:
- tests-mysql
- tests-postgres
- tests-kubernetes
- if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test'
+ if: >
+ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ||
+ github.ref == 'refs/heads/v2-0-test'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
diff --git a/BREEZE.rst b/BREEZE.rst
index 095fe1b..43705f9 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1188,7 +1188,12 @@ This is the current syntax for `./breeze <./breeze>`_:
image building time in production image and at container entering time for CI image. One of:
1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3
- 1.10.2 wheel
+ 1.10.2 wheel none
+
+ When 'none' is used, you can install airflow from local packages. When building image,
+ airflow package should be added to 'docker-context-files' and
+ --install-from-docker-context-files flag should be used. When running an image, airflow
+ package should be added to dist folder and --install-packages-from-dist flag should be used.
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
If specified, installs Airflow directly from reference in GitHub. This happens at
@@ -1198,10 +1203,10 @@ This is the current syntax for `./breeze <./breeze>`_:
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
- --install-wheels
- If specified it will look for wheel packages placed in dist folder and it will install the
- wheels from there after installing Airflow. This is useful for testing backport
- packages as well as in the future for testing provider packages for 2.0.
+ --install-packages-from-dist
+ If specified it will look for packages placed in dist folder and it will install the
+ packages after installing Airflow. This is useful for testing provider
+ packages.
-I, --production-image
Use production image for entering the environment and builds (not for tests).
@@ -1231,10 +1236,10 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --skip-installing-airflow-via-pip
- Skips installing Airflow via PIP. If you use this flag and want to install
- Airflow, you have to install Airflow from packages placed in
- 'docker-context-files' and use --add-local-pip-files flag.
+ --disable-pypi-when-building
+ Disable installing Airflow from pypi when building. If you use this flag and want
+ to install Airflow, you have to install it from packages placed in
+ 'docker-context-files' and use --install-from-local-files-when-building flag.
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -1287,13 +1292,13 @@ This is the current syntax for `./breeze <./breeze>`_:
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
--disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+ Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
- --add-local-pip-wheels
+ --install-from-local-files-when-building
This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl packages placed
+ Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl files
+ the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
@@ -1758,10 +1763,10 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --skip-installing-airflow-via-pip
- Skips installing Airflow via PIP. If you use this flag and want to install
- Airflow, you have to install Airflow from packages placed in
- 'docker-context-files' and use --add-local-pip-files flag.
+ --disable-pypi-when-building
+ Disable installing Airflow from pypi when building. If you use this flag and want
+ to install Airflow, you have to install it from packages placed in
+ 'docker-context-files' and use --install-from-local-files-when-building flag.
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -1814,13 +1819,13 @@ This is the current syntax for `./breeze <./breeze>`_:
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
--disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+ Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
- --add-local-pip-wheels
+ --install-from-local-files-when-building
This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl packages placed
+ Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl files
+ the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
@@ -2101,7 +2106,12 @@ This is the current syntax for `./breeze <./breeze>`_:
image building time in production image and at container entering time for CI image. One of:
1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3
- 1.10.2 wheel
+ 1.10.2 wheel none
+
+ When 'none' is used, you can install airflow from local packages. When building image,
+ airflow package should be added to 'docker-context-files' and
+ --install-from-docker-context-files flag should be used. When running an image, airflow
+ package should be added to dist folder and --install-packages-from-dist flag should be used.
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
If specified, installs Airflow directly from reference in GitHub. This happens at
@@ -2111,10 +2121,10 @@ This is the current syntax for `./breeze <./breeze>`_:
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
- --install-wheels
- If specified it will look for wheel packages placed in dist folder and it will install the
- wheels from there after installing Airflow. This is useful for testing backport
- packages as well as in the future for testing provider packages for 2.0.
+ --install-packages-from-dist
+ If specified it will look for packages placed in dist folder and it will install the
+ packages after installing Airflow. This is useful for testing provider
+ packages.
****************************************************************************************************
Credentials
@@ -2151,10 +2161,10 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
- --skip-installing-airflow-via-pip
- Skips installing Airflow via PIP. If you use this flag and want to install
- Airflow, you have to install Airflow from packages placed in
- 'docker-context-files' and use --add-local-pip-files flag.
+ --disable-pypi-when-building
+ Disable installing Airflow from pypi when building. If you use this flag and want
+ to install Airflow, you have to install it from packages placed in
+ 'docker-context-files' and use --install-from-local-files-when-building flag.
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -2207,13 +2217,13 @@ This is the current syntax for `./breeze <./breeze>`_:
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
--disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+ Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
- --add-local-pip-wheels
+ --install-from-local-files-when-building
This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl packages placed
+ Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl files
+ the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
@@ -2292,6 +2302,17 @@ This is the current syntax for `./breeze <./breeze>`_:
Default: All
****************************************************************************************************
+ Flags for generation of the packages
+
+ -S, --version-suffix-for-pypi SUFFIX
+ Adds optional suffix to the version in the generated backport package. It can be used
+ to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
+
+ -N, --version-suffix-for-svn SUFFIX
+ Adds optional suffix to the generated names of package. It can be used to generate
+ rc1/rc2 ... versions of the packages to be uploaded to SVN.
+
+ ****************************************************************************************************
Increase verbosity of the scripts
-v, --verbose
diff --git a/CI.rst b/CI.rst
index fac9f0f..e087438 100644
--- a/CI.rst
+++ b/CI.rst
@@ -730,15 +730,15 @@ The image names follow the patterns:
| | | <COMMIT_SHA> | It contains only compiled libraries and minimal set of dependencies to run Airflow. |
+--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+
-* <BRANCH> might be either "master" or "v1-10-test"
-* <X.Y> - Python version (Major + Minor). For "master" it should be in ["3.6", "3.7", "3.8"]. For
+* <BRANCH> might be either "master" or "v1-10-test" or "v2-0-test"
+* <X.Y> - Python version (Major + Minor). For "master" and "v2-0-test" should be in ["3.6", "3.7", "3.8"]. For
v1-10-test it should be in ["2.7", "3.5", "3.6". "3.7", "3.8"].
* <RUN_ID> - GitHub Actions RUN_ID. You can get it from CI action job outputs (run id is printed in
logs and displayed as part of the step name. All PRs belong to some RUN_ID and this way you can
pull the very exact version of image used in that RUN_ID
-* <COMMIT_SHA> - for images that get merged to "master" of "v1-10-test" the images are also tagged
+* <COMMIT_SHA> - for images that get merged to "master", "v2-0-test" of "v1-10-test" the images are also tagged
with the commit SHA of that particular commit. This way you can easily find the image that was used
- for testing for that "master" or "v1-10-test" test run.
+ for testing for that "master", "v2-0-test" or "v1-10-test" test run.
Reproducing CI Runs locally
===========================
diff --git a/breeze b/breeze
index d99ba02..e9a9557 100755
--- a/breeze
+++ b/breeze
@@ -57,7 +57,6 @@ export EXTRA_STATIC_CHECK_OPTIONS
# MAX_SCREEN_WIDTH
# SCREEN_WIDTH
# MOUNT_LOCAL_SOURCES
-# MOUNT_FILES
# FORCE_PULL_IMAGES
# ENABLE_KIND_CLUSTER
# FORWARD_CREDENTIALS
@@ -98,9 +97,6 @@ function breeze::setup_default_breeze_constants() {
# By default we mount local Airflow sources
export MOUNT_LOCAL_SOURCES="true"
- # By default we mount files folder
- export MOUNT_FILES="true"
-
# By default we only pull images if we do not have them locally.
# This can be overridden by '--force-pull-images' flag
export FORCE_PULL_IMAGES="false"
@@ -140,7 +136,7 @@ function breeze::setup_default_breeze_constants() {
# if set to true, the ci image will look for wheel packages in dist folder and will install them
# during entering the container
- export INSTALL_WHEELS="false"
+ export INSTALL_PACKAGES_FROM_DIST="false"
# Determines whether to force build without checking if it is needed
# Can be overridden by '--force-build-images' flag.
@@ -638,19 +634,14 @@ function breeze::prepare_command_files() {
local remove_sources_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/remove-sources.yml
local forward_credentials_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/forward-credentials.yml
- local compose_ci_file=${main_ci_docker_compose_file}:${backend_docker_compose_file}
- local compose_prod_file=${main_prod_docker_compose_file}:${backend_docker_compose_file}
+ local compose_ci_file=${main_ci_docker_compose_file}:${backend_docker_compose_file}:${files_docker_compose_file}
+ local compose_prod_file=${main_prod_docker_compose_file}:${backend_docker_compose_file}:${files_docker_compose_file}
if [[ "${MOUNT_LOCAL_SOURCES}" != "false" ]]; then
compose_ci_file=${compose_ci_file}:${local_docker_compose_file}
compose_prod_file=${compose_prod_file}:${local_prod_docker_compose_file}
fi
- if [[ "${MOUNT_FILES}" != "false" ]]; then
- compose_ci_file=${compose_ci_file}:${files_docker_compose_file}
- compose_prod_file=${compose_prod_file}:${files_docker_compose_file}
- fi
-
if [[ ${FORWARD_CREDENTIALS} == "true" ]]; then
compose_ci_file=${compose_ci_file}:${forward_credentials_docker_compose_file}
compose_prod_file=${compose_prod_file}:${forward_credentials_docker_compose_file}
@@ -940,10 +931,10 @@ function breeze::parse_arguments() {
echo
shift
;;
- --skip-installing-airflow-via-pip)
- export INSTALL_AIRFLOW_VIA_PIP="false"
+ --disable-pypi-when-building)
+ export INSTALL_FROM_PYPI="false"
export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
- echo "Skip installing airflow via PIP"
+ echo "Disable installing airflow from PYPI"
shift
;;
-E | --extras)
@@ -1027,8 +1018,8 @@ function breeze::parse_arguments() {
export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
shift
;;
- --add-local-pip-wheels)
- export AIRFLOW_LOCAL_PIP_WHEELS="true"
+ --install-from-local-files-when-building)
+ export INSTALL_FROM_DOCKER_CONTEXT_FILES="true"
echo "Install wheels from local docker-context-files when building image"
shift
;;
@@ -1057,7 +1048,7 @@ function breeze::parse_arguments() {
;;
-c | --github-registry)
echo
- echo "Use github registry"
+ echo "Use GitHub registry"
echo
export USE_GITHUB_REGISTRY="true"
shift
@@ -1066,7 +1057,7 @@ function breeze::parse_arguments() {
echo
echo "GitHub repository: ${2}"
echo
- echo "Using github registry."
+ echo "Using GitHub registry."
echo
export GITHUB_REPOSITORY="${2}"
export USE_GITHUB_REGISTRY="true"
@@ -1076,9 +1067,10 @@ function breeze::parse_arguments() {
echo
echo "GitHub image id: ${2}"
echo
- echo "Force pulling the image, using github registry and skip mounting local sources."
+ echo "Force pulling the image, using GitHub registry and skip mounting local sources."
echo "This is in order to get the exact same version as used in CI environment for SHA/RUN_ID!."
- echo "You can specify --skip-mounting-local-sources to not mount local sources. "
+ echo "You can specify --skip-mounting-local-sources to not mount local sources to get exact. "
+ echo "behaviour as in the CI environment."
echo
export FORCE_PULL_IMAGES="true"
export USE_GITHUB_REGISTRY="true"
@@ -1096,6 +1088,18 @@ function breeze::parse_arguments() {
echo
shift 2
;;
+ -S | --version-suffix-for-pypi)
+ export VERSION_SUFFIX_FOR_PYPI="${2}"
+ echo "Version suffix for PyPI ${VERSION_SUFFIX_FOR_PYPI}"
+ echo
+ shift 2
+ ;;
+ -N | --version-suffix-for-svn)
+ export VERSION_SUFFIX_FOR_SVN="${2}"
+ echo "Version suffix for SVN ${VERSION_SUFFIX_FOR_SVN}"
+ echo
+ shift 2
+ ;;
--load-example-dags)
export LOAD_EXAMPLES="true"
echo "Include Airflow sample dags"
@@ -1120,12 +1124,18 @@ function breeze::parse_arguments() {
echo
shift
;;
- --install-wheels)
- export INSTALL_WHEELS="true"
- echo "Install wheels found in dist folder during entering breeze."
+ --install-packages-from-dist)
+ export INSTALL_PACKAGES_FROM_DIST="true"
+ echo "Install packages found in dist folder when entering breeze."
echo
shift
;;
+ --package-format)
+ export PACKAGE_FORMAT="${2}"
+ echo "Selected package type: ${PACKAGE_FORMAT}"
+ echo
+ shift 2
+ ;;
--test-type)
export TEST_TYPE="${2}"
echo "Selected test type: ${TEST_TYPE}"
@@ -1201,6 +1211,12 @@ function breeze::parse_arguments() {
export UPGRADE_TO_LATEST_CONSTRAINTS="true"
shift
;;
+ prepare-airflow-packages)
+ last_subcommand="${1}"
+ command_to_run="perform_prepare_airflow_packages"
+ export INSTALL_PROVIDERS_FROM_SOURCES="false"
+ shift
+ ;;
push-image)
last_subcommand="${1}"
command_to_run="perform_push_image"
@@ -1432,9 +1448,13 @@ function breeze::prepare_formatted_versions() {
readonly FORMATTED_DEFAULT_PROD_EXTRAS
FORMATTED_TEST_TYPES=$(echo "${_breeze_allowed_test_types=""}" |
- tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
+ fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
readonly FORMATTED_TEST_TYPES
+ FORMATTED_PACKAGE_FORMATS=$(echo "${_breeze_allowed_package_formats=""}" |
+ fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
+ readonly FORMATTED_PACKAGE_FORMATS
+
}
#######################################################################################################
@@ -1470,6 +1490,8 @@ function breeze::prepare_usage() {
readonly USAGE_GENERATE_CONSTRAINTS
export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv"
readonly USAGE_INITIALIZE_LOCAL_VIRTUALENV
+ export USAGE_PREPARE_AIRFLOW_PACKAGES="Prepares airflow packages"
+ readonly USAGE_PREPARE_AIRFLOW_PACKAGES
export USAGE_PUSH_IMAGE="Pushes images to registry"
readonly USAGE_PUSH_IMAGE
export USAGE_KIND_CLUSTER="Manages KinD cluster on the host"
@@ -1641,6 +1663,23 @@ Flags:
$(breeze::flag_airflow_variants)
"
readonly DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV
+ export DETAILED_USAGE_PREPARE_AIRFLOW_PACKAGES="
+${CMDNAME} prepare-airflow-packages [FLAGS]
+
+ Prepares airflow packages (sdist and wheel) in dist folder. Note that
+ prepare-provider-packages command cleans up the dist folder, so if you want also
+ to generate provider packages, make sure you run prepare-provider-packages first,
+ and prepare-airflow-packages second.
+
+ General form:
+
+ '${CMDNAME} prepare-airflow-packages
+
+Flags:
+$(breeze::flag_packages)
+$(breeze::flag_verbosity)
+"
+ readonly DETAILED_USAGE_PREPARE_AIRFLOW_PACKAGES
export DETAILED_USAGE_PUSH_IMAGE="
${CMDNAME} push_image [FLAGS]
@@ -2124,6 +2163,11 @@ function breeze::flag_choose_different_airflow_version() {
${FORMATTED_INSTALL_AIRFLOW_VERSIONS}
+ When 'none' is used, you can install airflow from local packages. When building image,
+ airflow package should be added to 'docker-context-files' and
+ --install-from-docker-context-files flag should be used. When running an image, airflow
+ package should be added to dist folder and --install-packages-from-dist flag should be used.
+
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
If specified, installs Airflow directly from reference in GitHub. This happens at
image building time in production image and at container entering time for CI image.
@@ -2132,10 +2176,10 @@ ${FORMATTED_INSTALL_AIRFLOW_VERSIONS}
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
---install-wheels
- If specified it will look for wheel packages placed in dist folder and it will install the
- wheels from there after installing Airflow. This is useful for testing backport
- packages as well as in the future for testing provider packages for 2.0.
+--install-packages-from-dist
+ If specified it will look for packages placed in dist folder and it will install the
+ packages after installing Airflow. This is useful for testing provider
+ packages.
"
}
@@ -2176,6 +2220,29 @@ function breeze::flag_credentials() {
#######################################################################################################
#
+# Prints flags that control package preparation
+#
+# Outputs:
+# Flag information.
+#######################################################################################################
+function breeze::flag_packages() {
+ echo "
+--package-format PACKAGE_FORMAT
+
+ Chooses format of packages to prepare.
+
+ One of:
+
+${FORMATTED_PACKAGE_FORMATS}
+
+ Default: ${_breeze_default_package_formats:=}
+
+"
+}
+
+
+#######################################################################################################
+#
# Prints flags that control verbosity
#
# Outputs:
@@ -2244,10 +2311,10 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS}
--image-tag TAG
Additional tag in the image.
---skip-installing-airflow-via-pip
- Skips installing Airflow via PIP. If you use this flag and want to install
- Airflow, you have to install Airflow from packages placed in
- 'docker-context-files' and use --add-local-pip-files flag.
+--disable-pypi-when-building
+ Disable installing Airflow from pypi when building. If you use this flag and want
+ to install Airflow, you have to install it from packages placed in
+ 'docker-context-files' and use --install-from-local-files-when-building flag.
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -2300,13 +2367,13 @@ Build options:
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
--disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+ Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
---add-local-pip-wheels
+--install-from-local-files-when-building
This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl packages placed
+ Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl files
+ the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-C, --force-clean-images
Force build images with cache disabled. This will remove the pulled or build images
@@ -2392,6 +2459,24 @@ function breeze::flag_pull_push_docker_images() {
"
}
+#######################################################################################################
+#
+# Prints flags that control version of generated packages
+#
+# Outputs:
+# Flag information.
+#######################################################################################################
+function breeze::flag_version_suffix() {
+ echo "
+-S, --version-suffix-for-pypi SUFFIX
+ Adds optional suffix to the version in the generated backport package. It can be used
+ to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
+
+-N, --version-suffix-for-svn SUFFIX
+ Adds optional suffix to the generated names of package. It can be used to generate
+ rc1/rc2 ... versions of the packages to be uploaded to SVN.
+"
+}
#####################################################################################################
#
# Prints flags that control how Airflow should be populated with the command start-airflow
@@ -2406,7 +2491,6 @@ function breeze::flag_start_airflow() {
--load-default-connections
Include Airflow Default Connections.
-
"
}
@@ -2518,6 +2602,10 @@ $(breeze::print_star_line)
$(breeze::flag_tests)
$(breeze::print_star_line)
+ Flags for generation of the packages
+$(breeze::flag_version_suffix)
+
+$(breeze::print_star_line)
Increase verbosity of the scripts
$(breeze::flag_verbosity)
@@ -2659,6 +2747,7 @@ function breeze::check_and_save_all_params() {
parameters::check_and_save_allowed_param "MYSQL_VERSION" "Mysql version" "--mysql-version"
parameters::check_allowed_param TEST_TYPE "Type of tests" "--test-type"
+ parameters::check_allowed_param PACKAGE_FORMAT "Format of packages to build" "--package-format"
# Can't verify those - they can be anything, so let's just save them
@@ -2862,7 +2951,8 @@ function breeze::run_build_command() {
build_images::rebuild_ci_image_if_needed
fi
;;
- build_docs | perform_static_checks | perform_generate_constraints )
+ build_docs | perform_static_checks | perform_generate_constraints | \
+ perform_prepare_airflow_packages)
build_images::prepare_ci_build
build_images::rebuild_ci_image_if_needed
;;
@@ -2891,8 +2981,9 @@ function breeze::run_build_command() {
build_images::prepare_ci_build
fi
;;
- perform_initialize_local_virtualenv | perform_setup_autocomplete | toggle_suppress_cheatsheet | toggle_suppress_asciiart ) ;;
-
+ perform_initialize_local_virtualenv | perform_setup_autocomplete | \
+ toggle_suppress_cheatsheet | toggle_suppress_asciiart )
+ ;;
manage_kind_cluster)
if [[ ${KIND_CLUSTER_OPERATION} == "start" ]]; then
echo "Starts KinD cluster"
@@ -3014,6 +3105,9 @@ function breeze::run_breeze_command() {
perform_generate_constraints)
runs::run_generate_constraints
;;
+ perform_prepare_airflow_packages)
+ build_airflow_packages::build_airflow_packages
+ ;;
perform_push_image)
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
push_pull_remove_images::push_prod_images
diff --git a/breeze-complete b/breeze-complete
index 505c6bc..df81093 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -62,6 +62,7 @@ _breeze_allowed_install_airflow_versions=$(cat <<-EOF
1.10.3
1.10.2
wheel
+none
EOF
)
diff --git a/scripts/ci/constraints/ci_branch_constraints.sh b/scripts/ci/constraints/ci_branch_constraints.sh
index 4cc7227..6be2d84 100755
--- a/scripts/ci/constraints/ci_branch_constraints.sh
+++ b/scripts/ci/constraints/ci_branch_constraints.sh
@@ -22,6 +22,8 @@ if [[ ${GITHUB_REF} == 'refs/heads/master' ]]; then
echo "::set-output name=branch::constraints-master"
elif [[ ${GITHUB_REF} == 'refs/heads/v1-10-test' ]]; then
echo "::set-output name=branch::constraints-1-10"
+elif [[ ${GITHUB_REF} == 'refs/heads/v2-0-test' ]]; then
+ echo "::set-output name=branch::constraints-2-0"
else
echo
echo "Unexpected ref ${GITHUB_REF}. Exiting!"