[GitHub] [airflow] kaxil commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox <gi...@apache.org>]

kaxil commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334916160
 
 

 ##########
 File path: BREEZE.rst
 ##########
 @@ -46,409 +46,586 @@ Here is the short 10 minute video about Airflow Breeze
 Prerequisites
 =============
 
-Docker
-------
+Docker Community Edition
+------------------------
 
-You need latest stable Docker Community Edition installed and on the PATH. It should be
-configured to be able to run ``docker`` commands directly and not only via root user. Your user
-should be in the ``docker`` group. See `Docker installation guide <https://docs.docker.com/install/>`_
+- **Version**: Install the latest stable Docker Community Edition and add it to the PATH.
+- **Permissions**: Configure to run the ``docker`` commands directly and not only via root user.
+  Your user should be in the ``docker`` group.
+  See `Docker installation guide <https://docs.docker.com/install/>`_ for details.
+- **Disk space**: On macOS, increase your available disk space before starting to work with
+  the environment. At least 128 GB of free disk space is recommended. You can also get by with a
+  smaller space but make sure to clean up the Docker disk space periodically.
+  See also `Docker for Mac - Space <https://docs.docker.com/docker-for-mac/space>`_ for details
+  on increasing disk space available for Docker on Mac.
+- **Docker problems**: Sometimes it is not obvious that space is an issue when you run into
+  a problem with Docker. If you see a weird behaviour, try
+  `cleaning up the images <#cleaning-up-the-images>`_. Also see
+  `pruning <https://docs.docker.com/config/pruning/>`_ instructions from Docker.
+
+Docker Compose
+--------------
 
-When you develop on Mac OS you usually have not enough disk space for Docker if you start using it
-seriously. You should increase disk space available before starting to work with the environment.
-Usually you have weird problems of docker containers when you run out of Disk space. It might not be
-obvious that space is an issue. At least 128 GB of Disk space is recommended. You can also get by with smaller space but you should more
-often clean the docker disk space periodically.
+- **Version**: Install the latest stable Docker Compose and add it to the PATH.
+  See `Docker Compose Installation Guide <https://docs.docker.com/compose/install/>`_ for details.
 
-If you get into weird behaviour try `Cleaning up the images <#cleaning-up-the-images>`_.
+- **Permissions**: Configure to run the ``docker-compose`` command.
 
-See also `Docker for Mac - Space <https://docs.docker.com/docker-for-mac/space>`_ for details of increasing
-disk space available for Docker on Mac.
+Docker Images Used by Breeze
+----------------------------
 
-Docker compose
---------------
+For all development tasks, related integration tests and static code checks, we use Docker
+images maintained on the Docker Hub in the ``apache/airflow`` repository.
 
-Latest stable Docker Compose installed and on the PATH. It should be
-configured to be able to run ``docker-compose`` command.
-See `Docker compose installation guide <https://docs.docker.com/compose/install/>`_
+There are three images that we are currently managing:
 
-Getopt and gstat
-----------------
+* **Slim CI** image that is used for static code checks (size of ~500MB). Its tag follows the pattern
+  of ``<BRANCH>-python<PYTHON_VERSION>-ci-slim`` (for example, ``apache/airflow:master-python3.6-ci-slim``).
+  The image is built using the `<Dockerfile>`_ Dockerfile.
+* **Full CI image*** that is used for testing. It contains a lot more test-related installed software
+  (size of ~1GB). Its tag follows the pattern of ``<BRANCH>-python<PYTHON_VERSION>-ci``
+  (for example, ``apache/airflow:master-python3.6-ci``). The image is built using the
+  `<Dockerfile>`_ Dockerfile.
+* **Checklicense image** that is used during license check with the Apache RAT tool. It does not
+  require any of the dependencies that the two CI images need so it is built using a different Dockerfile
+  `<Dockerfile-checklicence>`_ and only contains Java + Apache RAT tool. The image is
+  labelled with ``checklicence``, for example: ``apache/airflow:checklicence``. No versioning is used for
+  the Checklicence image.
 
-* If you are on MacOS
+Before you run tests, enter the environment or run local static checks, the necessary local images should be
+pulled and built from Docker Hub. This happens automatically for the test environment but you need to
+manually trigger it for static checks as described in `Building the images <#bulding-the-images>`_
+and `Pulling the latest images <#pulling-the-latest-images>`_.
+The static checks will fail and inform what to do if the image is not yet built.
 
-  * you need gnu ``getopt`` and ``gstat`` to get Airflow Breeze running.
+Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some
+time. But for subsequent source code changes, no wait time is expected.
+However, changes to sensitive files like setup.py or Dockerfile will trigger a rebuild
+that may take more time though it is highly optimized to only rebuild what is needed.
 
-  * Typically you need to run ``brew install gnu-getopt coreutils`` and then follow instructions (you need to link the gnu getopt
-    version to become first on the PATH). Make sure to re-login after yoy make the suggested changes.
+In most cases, rebuilding an image requires network connectivity (for example, to download new
+dependencies). If you work offline and do not want to rebuild the images when needed, you can set the
+``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
+`Default behaviour for user interaction <#default-behaviour-for-user-interaction>`_ section.
 
-  * Then (with brew) link the gnu-getopt to become default as suggested by brew.
+See `Troubleshooting section <#troubleshooting>`_ for steps you can make to clean the environment.
 
-  * If you use bash, you should run this command (and re-login):
+Getopt and gstat
 
 Review comment:
   ```suggestion
   Prerequisites 
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services