You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/07/30 11:49:23 UTC
[GitHub] [airflow] potiuk opened a new pull request, #32948: Add more detailed instructions on how to check pre-release candidates
potiuk opened a new pull request, #32948:
URL: https://github.com/apache/airflow/pull/32948
We ask our contributors to test pre-release candidates for Airflow and providers, but not everyone knows that Breeze allows to easily mix&match versions of airflow and providers that can be installed when running `breeze shell` or `breeze start-airflow` - which helps to test release candidates manually very, very quickly.
This change adds more detailed explanation and examples showing how it can be done, and links to those instruction from our release announcements and release documentation.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #32948: Add more detailed instructions on how to check pre-release candidates
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #32948:
URL: https://github.com/apache/airflow/pull/32948#discussion_r1278557402
##########
TESTING.rst:
##########
@@ -1442,6 +1442,88 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre release packages
Review Comment:
```suggestion
Testing pre-release packages
```
##########
TESTING.rst:
##########
@@ -1442,6 +1442,88 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre release packages
Review Comment:
hope this does not hamper the links for https://github.com/apache/airflow/blob/main/TESTING.rst#testing-pre-release-packages
##########
TESTING.rst:
##########
@@ -1442,6 +1442,88 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre release packages
+----------------------------
+
+There are two ways how you can get airflow packages in ``dist`` folder - by building them from sources or
+downloading them from PyPI.
+
+.. note ::
+
+ Make sure you run ``rm dist/*`` before you start building packages or downloading them from PyPI because
+ the packages built there already are not removed manually.
+
+In order to build apache-airflow from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-managment prepare-airflow-package
+
+In order to build providers from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-managment prepare-provider-packages <PROVIDER_1> <PROVIDER_2> ... <PROVIDER_N>
+
+The packages are built in ``dist`` folder and the command will summarise what packages are available in the
+``dist`` folder after it finishes.
+
+If you want to download the packages from PyPI, you need to run the following command:
+
+.. code-block:: bash
+
+ pip download apache-airflow-providers-<PROVIDER_NAME>==X.Y.Zrc1 --dest dist --no-deps
+
+You can use it for both release and pre-release packages.
+
+
+Few examples below will explain how you can test pre-release (also release) packages.
+
+This one will download ``airflow`` and ``celery`` and ``kubernetes`` provider packages from PyPI and
+eventually start Airflow using the packages downloaded with the Celery Executor. It will also
+load example dags and default connections:
+
+.. code:: bash
+
+ rm dist/*
+ pip download apache-airflow==2.7.0rc1 --dest dist --no-deps
+ pip download apache-airflow-providers-cncf-kubernetes==7.4.0rc1 --dest dist --no-deps
+ pip download apache-airflow-providers-cncf-kubernetes==3.3.0rc1 --dest dist --no-deps
+ breeze start-airflow --mount-sources remove --use-packages-from-dist --executor CeleryExecutor --load-default-connections --load-example-dags
+
+
+This one will download ``airflow`` and ``celery`` and ``kubernetes`` provider packages from PyPI but build
Review Comment:
Maybe this one will not download ``airflow`` from PyPI?
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -416,9 +421,9 @@ EOF
Note, For RC2/3 you may refer to shorten vote period as agreed in mailing list [thread](https://lists.apache.org/thread/cv194w1fqqykrhswhmm54zy9gnnv6kgm).
-# Verify the release candidate by PMCs
+# Verify the release candidate by PMC members
-The PMCs should verify the releases in order to make sure the release is following the
+The PMC members should verify the releases in order to make sure the release is following the
Review Comment:
```suggestion
PMC members should verify the releases in order to make sure the release is following the
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #32948: Add more detailed instructions on how to check pre-release candidates
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #32948:
URL: https://github.com/apache/airflow/pull/32948#discussion_r1278565782
##########
TESTING.rst:
##########
@@ -1442,6 +1442,88 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre release packages
Review Comment:
Yes. Was afraid of that too :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vatsrahul1001 commented on a diff in pull request #32948: Add more detailed instructions on how to check pre-release candidates
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on code in PR #32948:
URL: https://github.com/apache/airflow/pull/32948#discussion_r1278568464
##########
TESTING.rst:
##########
@@ -1442,6 +1442,88 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre-release packages
+----------------------------
+
+There are two ways how you can get airflow packages in ``dist`` folder - by building them from sources or
+downloading them from PyPI.
+
+.. note ::
+
+ Make sure you run ``rm dist/*`` before you start building packages or downloading them from PyPI because
+ the packages built there already are not removed manually.
+
+In order to build apache-airflow from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-managment prepare-airflow-package
Review Comment:
Typo: should be release-management
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #32948: Add more detailed instructions on how to check pre-release candidates
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #32948:
URL: https://github.com/apache/airflow/pull/32948#discussion_r1278570600
##########
TESTING.rst:
##########
@@ -1442,6 +1442,88 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre-release packages
+----------------------------
+
+There are two ways how you can get airflow packages in ``dist`` folder - by building them from sources or
+downloading them from PyPI.
+
+.. note ::
+
+ Make sure you run ``rm dist/*`` before you start building packages or downloading them from PyPI because
+ the packages built there already are not removed manually.
+
+In order to build apache-airflow from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-managment prepare-airflow-package
Review Comment:
Right. Fixed. Also added one more example and updated some wording.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Adaverse commented on a diff in pull request #32948: Add more detailed instructions on how to check pre-release candidates
Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32948:
URL: https://github.com/apache/airflow/pull/32948#discussion_r1278580179
##########
TESTING.rst:
##########
@@ -1442,6 +1442,104 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
Review Comment:
```suggestion
turn the CI image of Breeze to install and start Airflow for both Airflow and provider packages - both,
```
##########
TESTING.rst:
##########
@@ -1442,6 +1442,104 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
Review Comment:
```suggestion
source tree from the container when starting it, the second one installs ``airflow`` and ``providers``
```
##########
TESTING.rst:
##########
@@ -1442,6 +1442,104 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre-release packages
+----------------------------
+
+There are two ways how you can get airflow packages in ``dist`` folder - by building them from sources or
+downloading them from PyPI.
+
+.. note ::
+
+ Make sure you run ``rm dist/*`` before you start building packages or downloading them from PyPI because
+ the packages built there already are not removed manually.
+
+In order to build apache-airflow from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-management prepare-airflow-package
+
+In order to build providers from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-management prepare-provider-packages <PROVIDER_1> <PROVIDER_2> ... <PROVIDER_N>
+
+The packages are built in ``dist`` folder and the command will summarise what packages are available in the
+``dist`` folder after it finishes.
+
+If you want to download the packages from PyPI, you need to run the following command:
+
+.. code-block:: bash
+
+ pip download apache-airflow-providers-<PROVIDER_NAME>==X.Y.Zrc1 --dest dist --no-deps
+
+You can use it for both release and pre-release packages.
+
+Examples of testing pre-release packages
+----------------------------------------
+
+Few examples below explain how you can test pre-release packages, and combine them with locally build
+and released packages.
+
+The following example downloads ``apache-airflow`` and ``celery`` and ``kubernetes`` provider packages from PyPI and
+eventually starts Airflow with the Celery Executor. It also load example dags and default connections:
+
+.. code:: bash
+
+ rm dist/*
+ pip download apache-airflow==2.7.0rc1 --dest dist --no-deps
+ pip download apache-airflow-providers-cncf-kubernetes==7.4.0rc1 --dest dist --no-deps
+ pip download apache-airflow-providers-cncf-kubernetes==3.3.0rc1 --dest dist --no-deps
+ breeze start-airflow --mount-sources remove --use-packages-from-dist --executor CeleryExecutor --load-default-connections --load-example-dags
+
+
+The following example downloads ``celery`` and ``kubernetes`` provider packages from PyPI, builds
+``apache-airflow`` package from the main sources and eventually starts Airflow with the Celery Executor.
+It also loads example dags and default connections:
+
+.. code:: bash
+
+ rm dist/*
+ breeze release-management prepare-airflow-package
+ pip download apache-airflow-providers-cncf-kubernetes==7.4.0rc1 --dest dist --no-deps
+ pip download apache-airflow-providers-cncf-kubernetes==3.3.0rc1 --dest dist --no-deps
+ breeze start-airflow --mount-sources remove --use-packages-from-dist --executor CeleryExecutor --load-default-connections --load-example-dags
+
+The following example builds ``celery``, ``kubernetes`` provider packages from PyPI, downloads 2.6.3 version
+of ``apache-airflow`` package from PyPI and eventually starts Airflow using default executor
+for the backend chosen (no example dags, no default connections):
+
+.. code:: bash
+
+ rm dist/*
+ pip download apache-airflow==2.6.3 --dest dist --no-deps
+ breeze release-management prepare-provider-packages celery cncf.kubernetes
+ breeze start-airflow --mount-sources remove --use-packages-from-dist
+
+You can mix and match packages from PyPI (final or pre-release candidates) with locally build packages. You
+can also choose which providers you install this way (the ``--remove-sources`` flag makes sure that airflow
Review Comment:
```suggestion
can also choose which providers to install this way since the ``--remove-sources`` flag makes sure that Airflow
```
##########
TESTING.rst:
##########
@@ -1442,6 +1442,104 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre-release packages
+----------------------------
+
+There are two ways how you can get airflow packages in ``dist`` folder - by building them from sources or
+downloading them from PyPI.
+
+.. note ::
+
+ Make sure you run ``rm dist/*`` before you start building packages or downloading them from PyPI because
+ the packages built there already are not removed manually.
+
+In order to build apache-airflow from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-management prepare-airflow-package
+
+In order to build providers from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-management prepare-provider-packages <PROVIDER_1> <PROVIDER_2> ... <PROVIDER_N>
+
+The packages are built in ``dist`` folder and the command will summarise what packages are available in the
+``dist`` folder after it finishes.
+
+If you want to download the packages from PyPI, you need to run the following command:
+
+.. code-block:: bash
+
+ pip download apache-airflow-providers-<PROVIDER_NAME>==X.Y.Zrc1 --dest dist --no-deps
+
+You can use it for both release and pre-release packages.
+
+Examples of testing pre-release packages
+----------------------------------------
+
+Few examples below explain how you can test pre-release packages, and combine them with locally build
+and released packages.
+
+The following example downloads ``apache-airflow`` and ``celery`` and ``kubernetes`` provider packages from PyPI and
+eventually starts Airflow with the Celery Executor. It also load example dags and default connections:
Review Comment:
```suggestion
eventually starts Airflow with the Celery Executor. It also loads example dags and default connections:
```
##########
TESTING.rst:
##########
@@ -1442,6 +1442,104 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre-release packages
+----------------------------
+
+There are two ways how you can get airflow packages in ``dist`` folder - by building them from sources or
Review Comment:
```suggestion
There are two ways how you can get Airflow packages in ``dist`` folder - by building them from sources or
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #32948: Add more detailed instructions on how to check pre-release candidates
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #32948:
URL: https://github.com/apache/airflow/pull/32948#discussion_r1278565852
##########
TESTING.rst:
##########
@@ -1442,6 +1442,88 @@ You can also run complete k8s tests with
This will create cluster, build images, deploy airflow run tests and finally delete clusters as single
command. It is the way it is run in our CI, you can also run such complete tests in parallel.
+Manually testing release candidate packages
+===========================================
+
+Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily
+turn the CI image of Breeze to install and start airflow for both Airflow and provider packages - both,
+packages that are built from sources and packages that are downloaded from PyPI when they are released
+there as release candidates.
+
+The way to test it is rather straightforward:
+
+1) Make sure that the packages - both ``airflow`` and ``providers`` are placed in the ``dist`` folder
+ of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter)
+
+2) You can run ```breeze shell`` or ``breeze start-airflow`` commands with adding the following flags -
+ ``--mount-sources remove`` and ``--use-packages-from-dist``. The first one removes the ``airflow``
+ source tree from the container when starting it, the second one installs ``airflow`` and ``provider``
+ packages from the ``dist`` folder when entering breeze.
+
+Testing pre release packages
+----------------------------
+
+There are two ways how you can get airflow packages in ``dist`` folder - by building them from sources or
+downloading them from PyPI.
+
+.. note ::
+
+ Make sure you run ``rm dist/*`` before you start building packages or downloading them from PyPI because
+ the packages built there already are not removed manually.
+
+In order to build apache-airflow from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-managment prepare-airflow-package
+
+In order to build providers from sources, you need to run the following command:
+
+.. code-block:: bash
+
+ breeze release-managment prepare-provider-packages <PROVIDER_1> <PROVIDER_2> ... <PROVIDER_N>
+
+The packages are built in ``dist`` folder and the command will summarise what packages are available in the
+``dist`` folder after it finishes.
+
+If you want to download the packages from PyPI, you need to run the following command:
+
+.. code-block:: bash
+
+ pip download apache-airflow-providers-<PROVIDER_NAME>==X.Y.Zrc1 --dest dist --no-deps
+
+You can use it for both release and pre-release packages.
+
+
+Few examples below will explain how you can test pre-release (also release) packages.
+
+This one will download ``airflow`` and ``celery`` and ``kubernetes`` provider packages from PyPI and
+eventually start Airflow using the packages downloaded with the Celery Executor. It will also
+load example dags and default connections:
+
+.. code:: bash
+
+ rm dist/*
+ pip download apache-airflow==2.7.0rc1 --dest dist --no-deps
+ pip download apache-airflow-providers-cncf-kubernetes==7.4.0rc1 --dest dist --no-deps
+ pip download apache-airflow-providers-cncf-kubernetes==3.3.0rc1 --dest dist --no-deps
+ breeze start-airflow --mount-sources remove --use-packages-from-dist --executor CeleryExecutor --load-default-connections --load-example-dags
+
+
+This one will download ``airflow`` and ``celery`` and ``kubernetes`` provider packages from PyPI but build
Review Comment:
```suggestion
This one will download ``celery`` and ``kubernetes`` provider packages from PyPI but build
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #32948: Add more detailed instructions on how to check pre-release candidates
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #32948:
URL: https://github.com/apache/airflow/pull/32948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org