You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/31 13:39:25 UTC
[GitHub] [airflow] VBhojawala opened a new pull request #13404: Rebase
VBhojawala opened a new pull request #13404:
URL: https://github.com/apache/airflow/pull/13404
Re-based pull request
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VBhojawala closed pull request #13404: Developers Quick Start Guide
Posted by GitBox <gi...@apache.org>.
VBhojawala closed pull request #13404:
URL: https://github.com/apache/airflow/pull/13404
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VBhojawala commented on a change in pull request #13404: Developers Quick Start Guide
Posted by GitBox <gi...@apache.org>.
VBhojawala commented on a change in pull request #13404:
URL: https://github.com/apache/airflow/pull/13404#discussion_r550522924
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
Review comment:
Sure, i will update it when it's merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13404: Developers Quick Start Guide
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13404:
URL: https://github.com/apache/airflow/pull/13404#discussion_r550493958
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
Review comment:
Maybe add a comment here that you will install airflow in the virtual env in the next step. It's not clear why empty env is created here.
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
Review comment:
```suggestion
It may requires some packages to be installed, watch the output of the command to see which ones are missing.
```
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
Review comment:
Remnant from merge. Should be removed.
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+- Running specific type of test
+
+ - Types of tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
+ All CLI Heisentests Integration Other Providers
+ API Core Helm MySQL Postgres WWW
+
+
+ - Running specific type of Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+
+
+- Running Integration test for specific test type
+
+
+ - Types of Integration Tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
+
+ all cassandra kerberos mongo openldap presto rabbitmq redis
+
+ - Running an Integration Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+
+
+- For more information on Testing visit : |TESTING.rst|
+
+.. |TESTING.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst" target="_blank">TESTING.rst</a>
+
+- Following are the some of important topics of TESTING.rst
+
+ - |Airflow Test Infrastructure|
+
+ .. |Airflow Test Infrastructure| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-test-infrastructure" target="_blank">
+ Airflow Test Infrastructure</a>
+
+
+ - |Airflow Unit Tests|
+
+ .. |Airflow Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-unit-tests" target="_blank">Airflow Unit
+ Tests</a>
+
+
+ - |Helm Unit Tests|
+
+ .. |Helm Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#helm-unit-tests" target="_blank">Helm Unit Tests
+ </a>
+
+
+ - |Airflow Integration Tests|
+
+ .. |Airflow Integration Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-integration-tests" target="_blank">
+ Airflow Integration Tests</a>
+
+
+ - |Running Tests with Kubernetes|
+
+ .. |Running Tests with Kubernetes| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes" target="_blank">
+ Running Tests with Kubernetes</a>
+
+
+ - |Airflow System Tests|
+
+ .. |Airflow System Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests" target="_blank">Airflow
+ System Tests</a>
+
+
+ - |Local and Remote Debugging in IDE|
+
+ .. |Local and Remote Debugging in IDE| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#local-and-remote-debugging-in-ide"
+ target="_blank">Local and Remote Debugging in IDE</a>
+
+
+ - |BASH Unit Testing (BATS)|
+
+ .. |BASH Unit Testing (BATS)| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#bash-unit-testing-bats" target="_blank">
+ BASH Unit Testing (BATS)</a>
+
+
+Pre-commit
+-----------------------------------
+
+Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
+such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
+tests are applied when you commit your code.
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:20px">
+ <img src="images/quick_start/ci_tests.png"
+ alt="CI tests Github">
+ </div>
+
+
+To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
+
+1. Installing required packages
+
+.. code-block:: bash
+
+ $ sudo apt install libxml2-utils
+
+2. Installing required Python packages
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install pre-commit
+
+3. Go to your project directory
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+
+
+4. Running pre-commit hooks
+
+.. code-block:: bash
+
+ $ pre-commit run --all-files
+ No-tabs checker......................................................Passed
+ Add license for all SQL files........................................Passed
+ Add license for all other files......................................Passed
+ Add license for all rst files........................................Passed
+ Add license for all JS/CSS/PUML files................................Passed
+ Add license for all JINJA template files.............................Passed
+ Add license for all shell files......................................Passed
+ Add license for all python files.....................................Passed
+ Add license for all XML files........................................Passed
+ Add license for all yaml files.......................................Passed
+ Add license for all md files.........................................Passed
+ Add license for all mermaid files....................................Passed
+ Add TOC for md files.................................................Passed
+ Add TOC for upgrade documentation....................................Passed
+ Check hooks apply to the repository..................................Passed
+ black................................................................Passed
+ Check for merge conflicts............................................Passed
+ Debug Statements (Python)............................................Passed
+ Check builtin type constructor use...................................Passed
+ Detect Private Key...................................................Passed
+ Fix End of Files.....................................................Passed
+ ...........................................................................
+
+5. Running pre-commit for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
+
+
+
+6. Running specific hook for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
+ black...............................................................Passed
+ $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
+ Run flake8..........................................................Passed
+
+
+
+<<<<<<< HEAD
Review comment:
Another merge remnant.
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+- Running specific type of test
+
+ - Types of tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
+ All CLI Heisentests Integration Other Providers
+ API Core Helm MySQL Postgres WWW
+
+
+ - Running specific type of Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+
+
+- Running Integration test for specific test type
+
+
+ - Types of Integration Tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
+
+ all cassandra kerberos mongo openldap presto rabbitmq redis
+
+ - Running an Integration Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+
+
+- For more information on Testing visit : |TESTING.rst|
+
+.. |TESTING.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst" target="_blank">TESTING.rst</a>
+
+- Following are the some of important topics of TESTING.rst
+
+ - |Airflow Test Infrastructure|
+
+ .. |Airflow Test Infrastructure| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-test-infrastructure" target="_blank">
+ Airflow Test Infrastructure</a>
+
+
+ - |Airflow Unit Tests|
+
+ .. |Airflow Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-unit-tests" target="_blank">Airflow Unit
+ Tests</a>
+
+
+ - |Helm Unit Tests|
+
+ .. |Helm Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#helm-unit-tests" target="_blank">Helm Unit Tests
+ </a>
+
+
+ - |Airflow Integration Tests|
+
+ .. |Airflow Integration Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-integration-tests" target="_blank">
+ Airflow Integration Tests</a>
+
+
+ - |Running Tests with Kubernetes|
+
+ .. |Running Tests with Kubernetes| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes" target="_blank">
+ Running Tests with Kubernetes</a>
+
+
+ - |Airflow System Tests|
+
+ .. |Airflow System Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests" target="_blank">Airflow
+ System Tests</a>
+
+
+ - |Local and Remote Debugging in IDE|
+
+ .. |Local and Remote Debugging in IDE| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#local-and-remote-debugging-in-ide"
+ target="_blank">Local and Remote Debugging in IDE</a>
+
+
+ - |BASH Unit Testing (BATS)|
+
+ .. |BASH Unit Testing (BATS)| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#bash-unit-testing-bats" target="_blank">
+ BASH Unit Testing (BATS)</a>
+
+
+Pre-commit
+-----------------------------------
+
+Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
+such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
+tests are applied when you commit your code.
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:20px">
+ <img src="images/quick_start/ci_tests.png"
+ alt="CI tests Github">
+ </div>
+
+
+To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
+
+1. Installing required packages
+
+.. code-block:: bash
+
+ $ sudo apt install libxml2-utils
+
+2. Installing required Python packages
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install pre-commit
+
+3. Go to your project directory
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+
+
+4. Running pre-commit hooks
+
+.. code-block:: bash
+
+ $ pre-commit run --all-files
+ No-tabs checker......................................................Passed
+ Add license for all SQL files........................................Passed
+ Add license for all other files......................................Passed
+ Add license for all rst files........................................Passed
+ Add license for all JS/CSS/PUML files................................Passed
+ Add license for all JINJA template files.............................Passed
+ Add license for all shell files......................................Passed
+ Add license for all python files.....................................Passed
+ Add license for all XML files........................................Passed
+ Add license for all yaml files.......................................Passed
+ Add license for all md files.........................................Passed
+ Add license for all mermaid files....................................Passed
+ Add TOC for md files.................................................Passed
+ Add TOC for upgrade documentation....................................Passed
+ Check hooks apply to the repository..................................Passed
+ black................................................................Passed
+ Check for merge conflicts............................................Passed
+ Debug Statements (Python)............................................Passed
+ Check builtin type constructor use...................................Passed
+ Detect Private Key...................................................Passed
+ Fix End of Files.....................................................Passed
+ ...........................................................................
+
+5. Running pre-commit for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
+
+
+
+6. Running specific hook for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
+ black...............................................................Passed
+ $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
+ Run flake8..........................................................Passed
+
+
+
+<<<<<<< HEAD
+7. Running specific checks in container using shell scripts. Scripts are located in ``.\scripts\in_container``
+ directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+
+
+
+8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+=======
+7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+ if found any error.
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+ $ pre-commit install
+ $ git commit -m "Added xyz"
+
+<<<<<<< HEAD
+9. To disable Pre-commit
+=======
+8. To disable Pre-commit
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+ $ pre-commit uninstall
+
+
+- For more information on visit |STATIC_CODE_CHECKS.rst|
+
+.. |STATIC_CODE_CHECKS.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst" target="_blank">
+ STATIC_CODE_CHECKS.rst</a>
+
+- Following are some of the important links of STATIC_CODE_CHECKS.rst
+
+ - |Pre-commit Hooks|
+
+ .. |Pre-commit Hooks| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#pre-commit-hooks" target="_blank">
+ Pre-commit Hooks</a>
+
+ - |Pylint Static Code Checks|
+
+ .. |Pylint Static Code Checks| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#pylint-static-code-checks"
+ target="_blank">Pylint Static Code Checks</a>
+
+
+ - |Running Static Code Checks via Breeze|
+
+ .. |Running Static Code Checks via Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#running-static-code-checks-via-breeze"
+ target="_blank">Running Static Code Checks via Breeze</a>
+
+
+
+
+
+Contribution guide
+-----------------------------------
+
+- To know how to contribute to the project visit |CONTRIBUTING.rst|
+
+.. |CONTRIBUTING.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst" target="_blank">CONTRIBUTING.rst</a>
+
+- Following are some of important links of CONTRIBUTING.rst
+
+ - |Types of contributions|
+
+ .. |Types of contributions| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions" target="_blank">
+ Types of contributions</a>
+
+
+ - |Roles of contributor|
+
+ .. |Roles of contributor| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#roles" target="_blank">Roles of
+ contributor</a>
+
+
+ - |Workflow for a contribution|
+
+ .. |Workflow for a contribution| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow" target="_blank">
+ Workflow for a contribution</a>
+
+
+
+Raising Pull Request
+-----------------------------------
+
+1. Go to your Github account and open your fork project and click on Branches
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pr1.png"
+ alt="Goto fork and select branches">
+ </div>
+
+2. Click on ``New pull request`` button on branch from which you want to raise a pull request.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pr2.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Add title and description as per Contributing guidelines and click on ``Create pull request``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pr3.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+Syncing Fork and rebasing Pull request
+--------------------------------------
+
Review comment:
Maybe one paragraph of explantation why this is needed? something like:
Often it takes several days or weeks to discuss and iterate with the PR until it is ready to merge. In the meantime new commits are merged, and you might run into conflicts, therefore you should periodically synchronize master in your fork with the `apache/airflow` master and rebase your PR on top of it. This chapter describes how to do it.
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
Review comment:
You will have to update that slightly (including the screenshot). We are just about to merge a change #13313 where we have additional window started that watches and recompiles assets automatically for developing webserver. This is a cool addition and it would be great if you add information about it here :).
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
Review comment:
```suggestion
6. Installing airflow in the local virtual environment ``airflow-env`` with breeze.
```
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+- Running specific type of test
+
+ - Types of tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
+ All CLI Heisentests Integration Other Providers
+ API Core Helm MySQL Postgres WWW
+
+
+ - Running specific type of Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+
+
+- Running Integration test for specific test type
+
+
+ - Types of Integration Tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
+
+ all cassandra kerberos mongo openldap presto rabbitmq redis
Review comment:
List of integrations changed in the meantime (added pinot).
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
Review comment:
Same here
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+- Running specific type of test
+
+ - Types of tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
+ All CLI Heisentests Integration Other Providers
+ API Core Helm MySQL Postgres WWW
+
+
+ - Running specific type of Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+
+
+- Running Integration test for specific test type
+
+
+ - Types of Integration Tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
+
+ all cassandra kerberos mongo openldap presto rabbitmq redis
+
+ - Running an Integration Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+
+
+- For more information on Testing visit : |TESTING.rst|
+
+.. |TESTING.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst" target="_blank">TESTING.rst</a>
+
+- Following are the some of important topics of TESTING.rst
+
+ - |Airflow Test Infrastructure|
+
+ .. |Airflow Test Infrastructure| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-test-infrastructure" target="_blank">
+ Airflow Test Infrastructure</a>
+
+
+ - |Airflow Unit Tests|
+
+ .. |Airflow Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-unit-tests" target="_blank">Airflow Unit
+ Tests</a>
+
+
+ - |Helm Unit Tests|
+
+ .. |Helm Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#helm-unit-tests" target="_blank">Helm Unit Tests
+ </a>
+
+
+ - |Airflow Integration Tests|
+
+ .. |Airflow Integration Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-integration-tests" target="_blank">
+ Airflow Integration Tests</a>
+
+
+ - |Running Tests with Kubernetes|
+
+ .. |Running Tests with Kubernetes| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes" target="_blank">
+ Running Tests with Kubernetes</a>
+
+
+ - |Airflow System Tests|
+
+ .. |Airflow System Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests" target="_blank">Airflow
+ System Tests</a>
+
+
+ - |Local and Remote Debugging in IDE|
+
+ .. |Local and Remote Debugging in IDE| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#local-and-remote-debugging-in-ide"
+ target="_blank">Local and Remote Debugging in IDE</a>
+
+
+ - |BASH Unit Testing (BATS)|
+
+ .. |BASH Unit Testing (BATS)| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#bash-unit-testing-bats" target="_blank">
+ BASH Unit Testing (BATS)</a>
+
+
+Pre-commit
+-----------------------------------
+
+Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
+such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
+tests are applied when you commit your code.
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:20px">
+ <img src="images/quick_start/ci_tests.png"
+ alt="CI tests Github">
+ </div>
+
+
+To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
+
+1. Installing required packages
+
+.. code-block:: bash
+
+ $ sudo apt install libxml2-utils
+
+2. Installing required Python packages
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install pre-commit
+
+3. Go to your project directory
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+
+
+4. Running pre-commit hooks
+
+.. code-block:: bash
+
+ $ pre-commit run --all-files
+ No-tabs checker......................................................Passed
+ Add license for all SQL files........................................Passed
+ Add license for all other files......................................Passed
+ Add license for all rst files........................................Passed
+ Add license for all JS/CSS/PUML files................................Passed
+ Add license for all JINJA template files.............................Passed
+ Add license for all shell files......................................Passed
+ Add license for all python files.....................................Passed
+ Add license for all XML files........................................Passed
+ Add license for all yaml files.......................................Passed
+ Add license for all md files.........................................Passed
+ Add license for all mermaid files....................................Passed
+ Add TOC for md files.................................................Passed
+ Add TOC for upgrade documentation....................................Passed
+ Check hooks apply to the repository..................................Passed
+ black................................................................Passed
+ Check for merge conflicts............................................Passed
+ Debug Statements (Python)............................................Passed
+ Check builtin type constructor use...................................Passed
+ Detect Private Key...................................................Passed
+ Fix End of Files.....................................................Passed
+ ...........................................................................
+
+5. Running pre-commit for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
+
+
+
+6. Running specific hook for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
+ black...............................................................Passed
+ $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
+ Run flake8..........................................................Passed
+
+
+
+<<<<<<< HEAD
+7. Running specific checks in container using shell scripts. Scripts are located in ``.\scripts\in_container``
+ directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+
+
+
+8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+=======
Review comment:
And here
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+- Running specific type of test
+
+ - Types of tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
+ All CLI Heisentests Integration Other Providers
+ API Core Helm MySQL Postgres WWW
+
+
+ - Running specific type of Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+
+
+- Running Integration test for specific test type
+
+
+ - Types of Integration Tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
+
+ all cassandra kerberos mongo openldap presto rabbitmq redis
+
+ - Running an Integration Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+
+
+- For more information on Testing visit : |TESTING.rst|
+
+.. |TESTING.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst" target="_blank">TESTING.rst</a>
+
+- Following are the some of important topics of TESTING.rst
+
+ - |Airflow Test Infrastructure|
+
+ .. |Airflow Test Infrastructure| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-test-infrastructure" target="_blank">
+ Airflow Test Infrastructure</a>
+
+
+ - |Airflow Unit Tests|
+
+ .. |Airflow Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-unit-tests" target="_blank">Airflow Unit
+ Tests</a>
+
+
+ - |Helm Unit Tests|
+
+ .. |Helm Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#helm-unit-tests" target="_blank">Helm Unit Tests
+ </a>
+
+
+ - |Airflow Integration Tests|
+
+ .. |Airflow Integration Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-integration-tests" target="_blank">
+ Airflow Integration Tests</a>
+
+
+ - |Running Tests with Kubernetes|
+
+ .. |Running Tests with Kubernetes| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes" target="_blank">
+ Running Tests with Kubernetes</a>
+
+
+ - |Airflow System Tests|
+
+ .. |Airflow System Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests" target="_blank">Airflow
+ System Tests</a>
+
+
+ - |Local and Remote Debugging in IDE|
+
+ .. |Local and Remote Debugging in IDE| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#local-and-remote-debugging-in-ide"
+ target="_blank">Local and Remote Debugging in IDE</a>
+
+
+ - |BASH Unit Testing (BATS)|
+
+ .. |BASH Unit Testing (BATS)| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#bash-unit-testing-bats" target="_blank">
+ BASH Unit Testing (BATS)</a>
+
+
+Pre-commit
+-----------------------------------
+
+Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
+such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
+tests are applied when you commit your code.
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:20px">
+ <img src="images/quick_start/ci_tests.png"
+ alt="CI tests Github">
+ </div>
+
+
+To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
+
+1. Installing required packages
+
+.. code-block:: bash
+
+ $ sudo apt install libxml2-utils
+
+2. Installing required Python packages
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install pre-commit
+
+3. Go to your project directory
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+
+
+4. Running pre-commit hooks
+
+.. code-block:: bash
+
+ $ pre-commit run --all-files
+ No-tabs checker......................................................Passed
+ Add license for all SQL files........................................Passed
+ Add license for all other files......................................Passed
+ Add license for all rst files........................................Passed
+ Add license for all JS/CSS/PUML files................................Passed
+ Add license for all JINJA template files.............................Passed
+ Add license for all shell files......................................Passed
+ Add license for all python files.....................................Passed
+ Add license for all XML files........................................Passed
+ Add license for all yaml files.......................................Passed
+ Add license for all md files.........................................Passed
+ Add license for all mermaid files....................................Passed
+ Add TOC for md files.................................................Passed
+ Add TOC for upgrade documentation....................................Passed
+ Check hooks apply to the repository..................................Passed
+ black................................................................Passed
+ Check for merge conflicts............................................Passed
+ Debug Statements (Python)............................................Passed
+ Check builtin type constructor use...................................Passed
+ Detect Private Key...................................................Passed
+ Fix End of Files.....................................................Passed
+ ...........................................................................
+
+5. Running pre-commit for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
+
+
+
+6. Running specific hook for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
+ black...............................................................Passed
+ $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
+ Run flake8..........................................................Passed
+
+
+
+<<<<<<< HEAD
+7. Running specific checks in container using shell scripts. Scripts are located in ``.\scripts\in_container``
+ directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+
+
+
+8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+=======
+7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
Review comment:
And here
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+- Running specific type of test
+
+ - Types of tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
+ All CLI Heisentests Integration Other Providers
+ API Core Helm MySQL Postgres WWW
+
+
+ - Running specific type of Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+
+
+- Running Integration test for specific test type
+
+
+ - Types of Integration Tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
+
+ all cassandra kerberos mongo openldap presto rabbitmq redis
+
+ - Running an Integration Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+
+
+- For more information on Testing visit : |TESTING.rst|
+
+.. |TESTING.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst" target="_blank">TESTING.rst</a>
+
+- Following are the some of important topics of TESTING.rst
+
+ - |Airflow Test Infrastructure|
+
+ .. |Airflow Test Infrastructure| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-test-infrastructure" target="_blank">
+ Airflow Test Infrastructure</a>
+
+
+ - |Airflow Unit Tests|
+
+ .. |Airflow Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-unit-tests" target="_blank">Airflow Unit
+ Tests</a>
+
+
+ - |Helm Unit Tests|
+
+ .. |Helm Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#helm-unit-tests" target="_blank">Helm Unit Tests
+ </a>
+
+
+ - |Airflow Integration Tests|
+
+ .. |Airflow Integration Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-integration-tests" target="_blank">
+ Airflow Integration Tests</a>
+
+
+ - |Running Tests with Kubernetes|
+
+ .. |Running Tests with Kubernetes| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes" target="_blank">
+ Running Tests with Kubernetes</a>
+
+
+ - |Airflow System Tests|
+
+ .. |Airflow System Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests" target="_blank">Airflow
+ System Tests</a>
+
+
+ - |Local and Remote Debugging in IDE|
+
+ .. |Local and Remote Debugging in IDE| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#local-and-remote-debugging-in-ide"
+ target="_blank">Local and Remote Debugging in IDE</a>
+
+
+ - |BASH Unit Testing (BATS)|
+
+ .. |BASH Unit Testing (BATS)| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#bash-unit-testing-bats" target="_blank">
+ BASH Unit Testing (BATS)</a>
+
+
+Pre-commit
+-----------------------------------
+
+Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
+such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
+tests are applied when you commit your code.
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:20px">
+ <img src="images/quick_start/ci_tests.png"
+ alt="CI tests Github">
+ </div>
+
+
+To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
+
+1. Installing required packages
+
+.. code-block:: bash
+
+ $ sudo apt install libxml2-utils
+
+2. Installing required Python packages
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install pre-commit
+
+3. Go to your project directory
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+
+
+4. Running pre-commit hooks
+
+.. code-block:: bash
+
+ $ pre-commit run --all-files
+ No-tabs checker......................................................Passed
+ Add license for all SQL files........................................Passed
+ Add license for all other files......................................Passed
+ Add license for all rst files........................................Passed
+ Add license for all JS/CSS/PUML files................................Passed
+ Add license for all JINJA template files.............................Passed
+ Add license for all shell files......................................Passed
+ Add license for all python files.....................................Passed
+ Add license for all XML files........................................Passed
+ Add license for all yaml files.......................................Passed
+ Add license for all md files.........................................Passed
+ Add license for all mermaid files....................................Passed
+ Add TOC for md files.................................................Passed
+ Add TOC for upgrade documentation....................................Passed
+ Check hooks apply to the repository..................................Passed
+ black................................................................Passed
+ Check for merge conflicts............................................Passed
+ Debug Statements (Python)............................................Passed
+ Check builtin type constructor use...................................Passed
+ Detect Private Key...................................................Passed
+ Fix End of Files.....................................................Passed
+ ...........................................................................
+
+5. Running pre-commit for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
+
+
+
+6. Running specific hook for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
+ black...............................................................Passed
+ $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
+ Run flake8..........................................................Passed
+
+
+
+<<<<<<< HEAD
+7. Running specific checks in container using shell scripts. Scripts are located in ``.\scripts\in_container``
+ directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+
+
+
+8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+=======
+7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+ if found any error.
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+ $ pre-commit install
+ $ git commit -m "Added xyz"
+
+<<<<<<< HEAD
+9. To disable Pre-commit
+=======
+8. To disable Pre-commit
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
Review comment:
Here too.
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+- Running specific type of test
+
+ - Types of tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
+ All CLI Heisentests Integration Other Providers
+ API Core Helm MySQL Postgres WWW
+
+
+ - Running specific type of Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+
+
+- Running Integration test for specific test type
+
+
+ - Types of Integration Tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
+
+ all cassandra kerberos mongo openldap presto rabbitmq redis
+
+ - Running an Integration Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+
+
+- For more information on Testing visit : |TESTING.rst|
+
+.. |TESTING.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst" target="_blank">TESTING.rst</a>
+
+- Following are the some of important topics of TESTING.rst
+
+ - |Airflow Test Infrastructure|
+
+ .. |Airflow Test Infrastructure| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-test-infrastructure" target="_blank">
+ Airflow Test Infrastructure</a>
+
+
+ - |Airflow Unit Tests|
+
+ .. |Airflow Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-unit-tests" target="_blank">Airflow Unit
+ Tests</a>
+
+
+ - |Helm Unit Tests|
+
+ .. |Helm Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#helm-unit-tests" target="_blank">Helm Unit Tests
+ </a>
+
+
+ - |Airflow Integration Tests|
+
+ .. |Airflow Integration Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-integration-tests" target="_blank">
+ Airflow Integration Tests</a>
+
+
+ - |Running Tests with Kubernetes|
+
+ .. |Running Tests with Kubernetes| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes" target="_blank">
+ Running Tests with Kubernetes</a>
+
+
+ - |Airflow System Tests|
+
+ .. |Airflow System Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests" target="_blank">Airflow
+ System Tests</a>
+
+
+ - |Local and Remote Debugging in IDE|
+
+ .. |Local and Remote Debugging in IDE| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#local-and-remote-debugging-in-ide"
+ target="_blank">Local and Remote Debugging in IDE</a>
+
+
+ - |BASH Unit Testing (BATS)|
+
+ .. |BASH Unit Testing (BATS)| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#bash-unit-testing-bats" target="_blank">
+ BASH Unit Testing (BATS)</a>
+
+
+Pre-commit
+-----------------------------------
+
+Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
+such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
+tests are applied when you commit your code.
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:20px">
+ <img src="images/quick_start/ci_tests.png"
+ alt="CI tests Github">
+ </div>
+
+
+To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
+
+1. Installing required packages
+
+.. code-block:: bash
+
+ $ sudo apt install libxml2-utils
+
+2. Installing required Python packages
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install pre-commit
+
+3. Go to your project directory
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+
+
+4. Running pre-commit hooks
+
+.. code-block:: bash
+
+ $ pre-commit run --all-files
+ No-tabs checker......................................................Passed
+ Add license for all SQL files........................................Passed
+ Add license for all other files......................................Passed
+ Add license for all rst files........................................Passed
+ Add license for all JS/CSS/PUML files................................Passed
+ Add license for all JINJA template files.............................Passed
+ Add license for all shell files......................................Passed
+ Add license for all python files.....................................Passed
+ Add license for all XML files........................................Passed
+ Add license for all yaml files.......................................Passed
+ Add license for all md files.........................................Passed
+ Add license for all mermaid files....................................Passed
+ Add TOC for md files.................................................Passed
+ Add TOC for upgrade documentation....................................Passed
+ Check hooks apply to the repository..................................Passed
+ black................................................................Passed
+ Check for merge conflicts............................................Passed
+ Debug Statements (Python)............................................Passed
+ Check builtin type constructor use...................................Passed
+ Detect Private Key...................................................Passed
+ Fix End of Files.....................................................Passed
+ ...........................................................................
+
+5. Running pre-commit for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
+
+
+
+6. Running specific hook for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
+ black...............................................................Passed
+ $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
+ Run flake8..........................................................Passed
+
+
+
+<<<<<<< HEAD
+7. Running specific checks in container using shell scripts. Scripts are located in ``.\scripts\in_container``
+ directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+
+
+
+8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+=======
+7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+ if found any error.
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+ $ pre-commit install
+ $ git commit -m "Added xyz"
+
+<<<<<<< HEAD
+9. To disable Pre-commit
+=======
Review comment:
Here too
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
+
+ Use CI image.
+
+ Branch name: master
+ Docker image: apache/airflow:master-python3.8-ci
+ Airflow source version: 2.0.0b2
+ Python version: 3.8
+ DockerHub user: apache
+ DockerHub repo: airflow
+ Backend: mysql 5.7
+
+
+ Port forwarding:
+
+ Ports are forwarded to the running docker containers for webserver and database
+ * 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 25555 -> forwarded to Flower dashboard -> airflow:5555
+ * 25433 -> forwarded to Postgres database -> postgres:5432
+ * 23306 -> forwarded to MySQL database -> mysql:3306
+ * 26379 -> forwarded to Redis broker -> redis:6379
+
+ Here are links to those services that you can use on host:
+ * Webserver: http://127.0.0.1:28080
+ * Flower: http://127.0.0.1:25555
+ * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
+ * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+ * Redis: redis://127.0.0.1:26379/0
+
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/start_airflow_tmux.png"
+ alt="Accessing local airflow">
+ </div>
+
+
+- Alternatively you can start the same using following commands
+
+ 1. Start Breeze
+
+ .. code-block:: bash
+
+ $ breeze --python 3.8 --backend mysql
+
+ 2. Open tmux
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# tmux
+
+ 3. Press Ctrl + B and "
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
+
+
+ 4. Press Ctrl + B and %
+
+ .. code-block:: bash
+
+ $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
+
+
+
+
+2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
+ and password ``admin``.
+
+ .. |http://127.0.0.1:28080| raw:: html
+
+ <a href="http://127.0.0.1:28080" target="_blank">http://127.0.0.1:28080</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/local_airflow.png"
+ alt="Accessing local airflow">
+ </div>
+
+3. Setup mysql database in mysql workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
+ blank(leave empty), default schema ``airflow``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/mysql_connection.png"
+ alt="Connecting to mysql">
+ </div>
+
+4. Stopping breeze
+
+.. code-block:: bash
+
+ root@f3619b74c59a:/opt/airflow# ./stop_airflow.sh
+ root@f3619b74c59a:/opt/airflow# exit
+ $ breeze stop
+
+5. Knowing more about Breeze
+
+.. code-block:: bash
+
+ $ breeze --help
+
+
+For more information visit : |Breeze documentation|
+
+.. |Breeze documentation| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze documentation</a>
+
+Following are some of important topics of Breeze documentation:
+
+
+- |Choosing different Breeze environment configuration|
+
+.. |Choosing different Breeze environment configuration| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#choosing-different-breeze-environment-configuration
+ " target="_blank">Choosing different Breeze environment configuration</a>
+
+
+- |Troubleshooting Breeze environment|
+
+.. |Troubleshooting Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#troubleshooting" target="_blank">Troubleshooting
+ Breeze environment</a>
+
+
+- |CLIs for cloud providers|
+
+.. |CLIs for cloud providers| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#clis-for-cloud-providers" target="_blank">CLIs for
+ cloud providers</a>
+
+
+- |Internal details of Breeze|
+
+.. |Internal details of Breeze| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#internal-details-of-breeze" target="_blank">
+ Internal details of Breeze</a>
+
+
+- |Breeze Command-Line Interface Reference|
+
+.. |Breeze Command-Line Interface Reference| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#breeze-command-line-interface-reference"
+ target="_blank">Breeze Command-Line Interface Reference</a>
+
+
+- |Cleaning the environment|
+
+.. |Cleaning the environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#cleaning-the-environment" target="_blank">
+ Cleaning the environment</a>
+
+
+- |Other uses of the Airflow Breeze environment|
+
+.. |Other uses of the Airflow Breeze environment| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst#other-uses-of-the-airflow-breeze-environment"
+ target="_blank">Other uses of the Airflow Breeze environment</a>
+
+
+
+Setting up Debug
+-----------------------------------------
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use sqlite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySql connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG in PyCharm
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add Interpreter.png"
+ alt="Adding existing interpreter">
+ </div>
+
+- In PyCharm IDE open airflow project, directory ``\files\dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in airflow project's ``\airflow\example_dags`` directory to ``\files\dags\``.
+
+- Add main block at the end of your DAG file to make it runnable. It will run a back_fill job:
+
+ .. code-block:: python
+
+ if __name__ == '__main__':
+ from airflow.utils.state import State
+ dag.clear(dag_run_state=State.NONE)
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_configuration.png"
+ alt="Add Configuration pycharm">
+ </div>
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/add_env_variable.png"
+ alt="Add environment variable pycharm">
+ </div>
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+
+
+Starting development
+###################################
+
+
+Creating a branch
+-----------------------------------
+
+1. Click on branch symbol in the bottom right corner of Pycharm
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/creating_branch_2.png"
+ alt="Giving name to a branch">
+ </div>
+
+
+
+Testing
+-----------------------------------
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
+
+ ======================================== 3 passed in 1.49s ========================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+<<<<<<< HEAD
+=======
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``.\scripts\in_container`` directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+
+- Running specific type of test
+
+ - Types of tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
+ All CLI Heisentests Integration Other Providers
+ API Core Helm MySQL Postgres WWW
+
+
+ - Running specific type of Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
+
+
+- Running Integration test for specific test type
+
+
+ - Types of Integration Tests
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
+
+ all cassandra kerberos mongo openldap presto rabbitmq redis
+
+ - Running an Integration Test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+
+
+- For more information on Testing visit : |TESTING.rst|
+
+.. |TESTING.rst| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst" target="_blank">TESTING.rst</a>
+
+- Following are the some of important topics of TESTING.rst
+
+ - |Airflow Test Infrastructure|
+
+ .. |Airflow Test Infrastructure| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-test-infrastructure" target="_blank">
+ Airflow Test Infrastructure</a>
+
+
+ - |Airflow Unit Tests|
+
+ .. |Airflow Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-unit-tests" target="_blank">Airflow Unit
+ Tests</a>
+
+
+ - |Helm Unit Tests|
+
+ .. |Helm Unit Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#helm-unit-tests" target="_blank">Helm Unit Tests
+ </a>
+
+
+ - |Airflow Integration Tests|
+
+ .. |Airflow Integration Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-integration-tests" target="_blank">
+ Airflow Integration Tests</a>
+
+
+ - |Running Tests with Kubernetes|
+
+ .. |Running Tests with Kubernetes| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes" target="_blank">
+ Running Tests with Kubernetes</a>
+
+
+ - |Airflow System Tests|
+
+ .. |Airflow System Tests| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests" target="_blank">Airflow
+ System Tests</a>
+
+
+ - |Local and Remote Debugging in IDE|
+
+ .. |Local and Remote Debugging in IDE| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#local-and-remote-debugging-in-ide"
+ target="_blank">Local and Remote Debugging in IDE</a>
+
+
+ - |BASH Unit Testing (BATS)|
+
+ .. |BASH Unit Testing (BATS)| raw:: html
+
+ <a href="https://github.com/apache/airflow/blob/master/TESTING.rst#bash-unit-testing-bats" target="_blank">
+ BASH Unit Testing (BATS)</a>
+
+
+Pre-commit
+-----------------------------------
+
+Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
+such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
+tests are applied when you commit your code.
+
+.. raw:: html
+
+ <div align="center" style="padding-bottom:20px">
+ <img src="images/quick_start/ci_tests.png"
+ alt="CI tests Github">
+ </div>
+
+
+To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
+
+1. Installing required packages
+
+.. code-block:: bash
+
+ $ sudo apt install libxml2-utils
+
+2. Installing required Python packages
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install pre-commit
+
+3. Go to your project directory
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+
+
+4. Running pre-commit hooks
+
+.. code-block:: bash
+
+ $ pre-commit run --all-files
+ No-tabs checker......................................................Passed
+ Add license for all SQL files........................................Passed
+ Add license for all other files......................................Passed
+ Add license for all rst files........................................Passed
+ Add license for all JS/CSS/PUML files................................Passed
+ Add license for all JINJA template files.............................Passed
+ Add license for all shell files......................................Passed
+ Add license for all python files.....................................Passed
+ Add license for all XML files........................................Passed
+ Add license for all yaml files.......................................Passed
+ Add license for all md files.........................................Passed
+ Add license for all mermaid files....................................Passed
+ Add TOC for md files.................................................Passed
+ Add TOC for upgrade documentation....................................Passed
+ Check hooks apply to the repository..................................Passed
+ black................................................................Passed
+ Check for merge conflicts............................................Passed
+ Debug Statements (Python)............................................Passed
+ Check builtin type constructor use...................................Passed
+ Detect Private Key...................................................Passed
+ Fix End of Files.....................................................Passed
+ ...........................................................................
+
+5. Running pre-commit for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
+
+
+
+6. Running specific hook for selected files
+
+.. code-block:: bash
+
+ $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
+ black...............................................................Passed
+ $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
+ Run flake8..........................................................Passed
+
+
+
+<<<<<<< HEAD
+7. Running specific checks in container using shell scripts. Scripts are located in ``.\scripts\in_container``
+ directory.
+
+.. code-block:: bash
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/
+ _in_container_script_init.sh run_generate_constraints.sh
+ check_environment.sh run_init_script.sh
+ entrypoint_ci.sh run_mypy.sh
+ entrypoint_exec.sh run_prepare_provider_packages.sh
+ prod/ run_prepare_provider_readme.sh
+ refresh_pylint_todo.sh run_pylint.sh
+ run_ci_tests.sh run_system_tests.sh
+ run_clear_tmp.sh run_test_package_import_all_classes.sh
+ run_cli_tool.sh run_test_package_install.sh
+ run_docs_build.sh run_tmux.sh
+ run_extract_tests.sh run_tmux_welcome.sh
+ run_fix_ownership.sh stop_tmux_airflow.sh
+ run_flake8.sh update_quarantined_test_status.py
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+
+
+
+8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+=======
+7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+>>>>>>> Moved contributrs-quick-start.rst to root directory and added testing scirpts example.
+ if found any error.
+
+.. code-block:: bash
+
+ $ cd ~/PycharmProjects/airflow
+ $ pre-commit install
+ $ git commit -m "Added xyz"
+
+<<<<<<< HEAD
Review comment:
Here too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13404: Developers Quick Start Guide
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13404:
URL: https://github.com/apache/airflow/pull/13404#discussion_r550570851
##########
File path: contributrs-quick-start.rst
##########
@@ -0,0 +1,976 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+*************************
+Contributor's Quick Guide
+*************************
+
+.. contents:: :local:
+
+
+Prerequisites
+#############
+
+1. Docker Community Edition
+2. Docker Compose
+3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
+
+
+Installing Prerequisites on Ubuntu
+###################################
+
+
+Docker Community Edition
+-----------------------------------
+
+
+1. Installing required packages for Docker and setting up docker repo
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+
+ $ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+ $ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable"
+
+2. Install Docker
+
+.. code-block:: bash
+
+ $ sudo apt-get update
+ $ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+3. Creating group for docker and adding current user to it.
+
+.. code-block:: bash
+
+ $ sudo groupadd docker
+ $ sudo usermod -aG docker $USER
+
+Note : After adding user to docker group Logout and Login again for group membership re-evaluation.
+
+4. Test Docker installation
+
+.. code-block:: bash
+
+ $ docker run hello-world
+
+
+
+
+Docker Compose
+--------------------------
+
+1. Installing latest version of Docker Compose
+
+.. code-block:: bash
+
+ $ COMPOSE_VERSION="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep '"tag_name":'\
+ | cut -d '"' -f 4)"
+
+ $ COMPOSE_URL="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/\
+ docker-compose-$(uname -s)-$(uname -m)"
+
+ $ sudo curl -L "${COMPOSE_URL}" -o /usr/local/bin/docker-compose
+
+ $ sudo chmod +x /usr/local/bin/docker-compose
+
+2. Verifying installation
+
+.. code-block:: bash
+
+ $ docker-compose --version
+
+
+
+pyenv and setting up virtual-env
+--------------------------------------------
+
+1. Checking required packages
+
+.. code-block:: bash
+
+ $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
+ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+
+ $ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
+ sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
+
+2. Install pyenv
+
+.. code-block:: bash
+
+ $ curl https://pyenv.run | bash
+
+3. Add the lines suggested at the end of installation to ~/.bashrc
+
+4. Restart your shell so the path changes take effect and verifying installation
+
+.. code-block:: bash
+
+ $ exec $SHELL
+ $ pyenv --version
+
+5. Checking available version, installing required Python version to pyenv and verifying it
+
+.. code-block:: bash
+
+ $ pyenv install --list
+ $ pyenv install 3.8.5
+ $ pyenv versions
+
+6. Creating new virtual environment named ``airflow-env`` for installed version python
+
+.. code-block:: bash
+
+ $ pyenv virtualenv 3.8.5 airflow-env
+
+7. Entering virtual environment ``airflow-env``
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+
+
+
+
+Setup Airflow with Breeze and PyCharm
+#####################################################
+
+
+Forking and cloning Project
+---------------------------
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+
+
+3. Open PyCharm and click ``Get from Version Control``
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/pycharm_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+4. Paste the copied clone link in URL and click on clone.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/click_on_clone.png"
+ alt="Cloning github fork to Pycharm">
+ </div>
+
+
+Setting up Breeze
+-----------------
+1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+
+.. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ cd ~/PycharmProjects/airflow/
+
+2. Initializing breeze autocomplete
+
+.. code-block:: bash
+
+ $ ./breeze setup-autocomplete
+ $ source ~/.bash_completion.d/breeze-complete
+
+3. Initialize breeze environment with required python version and backend. This may take a while for first time.
+
+.. code-block:: bash
+
+ $ ./breeze --python 3.8 --backend mysql
+
+4. Creating airflow tables and users. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get
+ the database/tables created.
+
+.. code-block:: bash
+
+ $ airflow db reset
+ $ airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname\
+ foo --lastname bar
+
+
+5. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+6. Linking Local virtual environment ``airflow-env`` with breeze.
+
+ It may requires some packages to be installed.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./breeze initialize-local-virtualenv --python 3.8
+
+
+7. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/PycharmProjects/airflow"
+ source ~/.bashrc
+
+Using Breeze
+-----------------------------------------
+
+1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
+ In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
+ It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
+ and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
+
+.. code-block:: bash
+
+ $ breeze start-airflow
Review comment:
Just merged! Please rebase and update and it will be ready to merge :).
Happy New Year!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org