You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/25 12:14:49 UTC
[GitHub] [airflow] j143 commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
j143 commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756838115
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1818,3 +1818,192 @@ describes how to do it.
.. raw:: html
</details>
+
+
+Setup and develop using Gitpod online workspaces
+################################################
+
+.. raw:: html
+
+ <details>
+ <summary>Setup and develop using Gitpod online workspaces</summary>
+
+
+
+Setup Airflow with Breeze
+-------------------------
+
+
+Forking and cloning Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+3. Add goto https://gitpod.io/#<copied-url> as shown.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_gitpod_url.png"
+ alt="Open personal airflow clone with Gitpod">
+ </div>
+
+Setting up Breeze
+~~~~~~~~~~~~~~~~~
+
+1. Breeze is already initialized in one of the terminals in Gitpod
+
+2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
+ is required to execute at least once for Airflow Breeze to get the database/tables created.
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow# airflow db reset
+ root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
+ --email admin@example.com --firstname foo --lastname bar
+
+
+3. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+
+Installing airflow with breeze.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Gitpod default image have all the required packages installed.
+
+1. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/Projects/airflow"
+ source ~/.bashrc
+
+
+Starting development
+--------------------
+
+
+Creating a branch
+~~~~~~~~~~~~~~~~~
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_2.png"
+ alt="Giving a name to a branch">
+ </div>
+
+
+
+Testing
+~~~~~~~
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# pytest tests/utils/test_session.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, cov-3.0.0, forked-1.3.0, httpx-0.15.0, instafail-0.4.2, rerunfailures-9.1.1, timeouts-1.2.1, xdist-2.4.0, requests-mock-1.9.3
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 4 items
+
+ tests/utils/test_session.py::TestSession::test_raised_provide_session PASSED [ 25%]
+ tests/utils/test_session.py::TestSession::test_provide_session_without_args_and_kwargs PASSED [ 50%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_args PASSED [ 75%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_kwargs PASSED [100%]
+
+====================================== 4 passed, 11 warnings in 33.14s ======================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``./scripts/in_container`` directory.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# ls ./scripts/in_container/
+ _in_container_script_init.sh quarantine_issue_header.md run_mypy.sh
+ _in_container_utils.sh run_anything.sh run_prepare_airflow_packages.sh
+ airflow_ci.cfg run_ci_tests.sh run_prepare_provider_documentation.sh
+ bin run_docs_build.sh run_prepare_provider_packages.sh
+ check_environment.sh run_extract_tests.sh run_resource_check.sh
+ check_junitxml_result.py run_fix_ownership.sh run_system_tests.sh
+ configure_environment.sh run_flake8.sh run_tmux_welcome.sh
+ entrypoint_ci.sh run_generate_constraints.sh stop_tmux_airflow.sh
+ entrypoint_exec.sh run_init_script.sh update_quarantined_test_status.py
+ prod run_install_and_test_provider_packages.sh
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+- Running specific type of test
+
+ - Types of tests
+
+ - Running specific type of test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Review comment:
This command gives an error for me.
<details>
<summary>error log</summary>
```c
gitpod /workspace/airflow $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Good version of docker 20.10.8.
Backend: mysql
MySQL version: 5.7
Python version: 3.8
Resetting the DB!
Selected test type: Core
mkdir: created directory '/workspace/airflow/.build/main/3.8'
mkdir: created directory '/workspace/airflow/.build/main/3.8/CI'
a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362
a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362
Use CI image.
Branch name: main
Docker image: ghcr.io/apache/airflow/main/ci/python3.8:latest
Airflow source version: 2.3.0.dev0
Python version: 3.8
Backend: mysql 5.7
####################################################################################################
Airflow Breeze CHEATSHEET
/workspace/airflow/breeze
####################################################################################################
Port forwarding:
Ports are forwarded to the running docker containers for webserver and database
* 12322 -> forwarded to Airflow ssh server -> airflow:22
* 28080 -> forwarded to Airflow webserver -> airflow:8080
* 25555 -> forwarded to Flower dashboard -> airflow:5555
* 25433 -> forwarded to Postgres database -> postgres:5432
* 23306 -> forwarded to MySQL database -> mysql:3306
* 21433 -> forwarded to MSSQL database -> mssql:1443
* 26379 -> forwarded to Redis broker -> redis:6379
Here are links to those services that you can use on host:
* ssh connection for remote debugging: ssh -p 12322 airflow@127.0.0.1 pw: airflow
* Webserver: http://127.0.0.1:28080
* Flower: http://127.0.0.1:25555
* Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
* Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
* Redis: redis://127.0.0.1:26379/0
####################################################################################################
You can setup autocomplete by running 'breeze setup-autocomplete'
####################################################################################################
You can toggle ascii/cheatsheet by running:
* breeze toggle-suppress-cheatsheet
* breeze toggle-suppress-asciiart
####################################################################################################
Unable to find image 'ghcr.io/apache/airflow/main/ci/python3.8:latest' locally
ylatest: Pulling from apache/airflow/main/ci/python3.8
a10c77af2613: Already exists
811275ca69b8: Already exists
a6e65fb20e69: Pulling fs layer
fc5582d0981c: Pulling fs layer
9c7a8f0f8e58: Pulling fs layer
8441add4ab92: Pulling fs layer
ebda9c5a1db7: Pulling fs layer
dbad869b28d4: Pulling fs layer
2a66638f1e61: Pulling fs layer
41837b664714: Pulling fs layer
a771646b27d5: Pulling fs layer
afb4979590d9: Pulling fs layer
428b7c9025e2: Pulling fs layer
bb98aff13f15: Pulling fs layer
9cd5fdf5c4d5: Pulling fs layer
6899d37becd2: Pulling fs layer
58f7721b779c: Pulling fs layer
b20ca6be30aa: Pulling fs layer
8441add4ab92: Waiting
ebda9c5a1db7: Waiting
6f8d52771bf7: Pulling fs layer
9a075d558b4b: Pulling fs layer
428b7c9025e2: Waiting
6ba6b39b87e7: Pulling fs layer
fa5244c3e2fa: Pulling fs layer
72edde940830: Pulling fs layer
2a66638f1e61: Waiting
afb4979590d9: Waiting
41837b664714: Waiting
c618f323bf15: Pulling fs layer
6f8d52771bf7: Waiting
9cd5fdf5c4d5: Waiting
6ba6b39b87e7: Waiting
bb98aff13f15: Waiting
478e02023d40: Pulling fs layer
58f7721b779c: Waiting
d0e12a8666f5: Pulling fs layer
fa5244c3e2fa: Waiting
9ba82d4ac0e8: Pulling fs layer
c618f323bf15: Waiting
478e02023d40: Waiting
83f21b096d92: Pulling fs layer
b20ca6be30aa: Waiting
72edde940830: Waiting
d0e12a8666f5: Waiting
6899d37becd2: Waiting
7e847a87dea6: Pulling fs layer
d0bd026e9517: Pulling fs layer
9a075d558b4b: Waiting
f6b0b6c17afd: Pulling fs layer
7e847a87dea6: Waiting
f5d244d620fb: Pulling fs layer
d0bd026e9517: Waiting
f6b0b6c17afd: Waiting
3f16cf0cdc03: Pulling fs layer
f5d244d620fb: Waiting
fc5582d0981c: Verifying Checksum
fc5582d0981c: Download complete
9c7a8f0f8e58: Verifying Checksum
9c7a8f0f8e58: Download complete
a6e65fb20e69: Verifying Checksum
a6e65fb20e69: Download complete
8441add4ab92: Verifying Checksum
8441add4ab92: Download complete
dbad869b28d4: Verifying Checksum
dbad869b28d4: Download complete
41837b664714: Verifying Checksum
41837b664714: Download complete
2a66638f1e61: Verifying Checksum
2a66638f1e61: Download complete
a6e65fb20e69: Pull complete
fc5582d0981c: Pull complete
afb4979590d9: Verifying Checksum
afb4979590d9: Download complete
428b7c9025e2: Verifying Checksum
428b7c9025e2: Download complete
9c7a8f0f8e58: Pull complete
bb98aff13f15: Verifying Checksum
bb98aff13f15: Download complete
a771646b27d5: Verifying Checksum
a771646b27d5: Download complete
6899d37becd2: Verifying Checksum
6899d37becd2: Download complete
8441add4ab92: Pull complete
58f7721b779c: Verifying Checksum
58f7721b779c: Download complete
ebda9c5a1db7: Download complete
b20ca6be30aa: Download complete
9a075d558b4b: Verifying Checksum
9a075d558b4b: Download complete
6ba6b39b87e7: Download complete
fa5244c3e2fa: Verifying Checksum
fa5244c3e2fa: Download complete
6f8d52771bf7: Verifying Checksum
6f8d52771bf7: Download complete
72edde940830: Verifying Checksum
72edde940830: Download complete
c618f323bf15: Verifying Checksum
c618f323bf15: Download complete
478e02023d40: Verifying Checksum
478e02023d40: Download complete
d0e12a8666f5: Verifying Checksum
d0e12a8666f5: Download complete
83f21b096d92: Download complete
9ba82d4ac0e8: Verifying Checksum
9ba82d4ac0e8: Download complete
7e847a87dea6: Download complete
d0bd026e9517: Verifying Checksum
d0bd026e9517: Download complete
f6b0b6c17afd: Verifying Checksum
f6b0b6c17afd: Download complete
f5d244d620fb: Verifying Checksum
f5d244d620fb: Download complete
3f16cf0cdc03: Verifying Checksum
3f16cf0cdc03: Download complete
9cd5fdf5c4d5: Verifying Checksum
9cd5fdf5c4d5: Download complete
ebda9c5a1db7: Pull complete
dbad869b28d4: Pull complete
2a66638f1e61: Pull complete
41837b664714: Pull complete
a771646b27d5: Pull complete
afb4979590d9: Pull complete
428b7c9025e2: Pull complete
bb98aff13f15: Pull complete
9cd5fdf5c4d5: Pull complete
6899d37becd2: Pull complete
58f7721b779c: Pull complete
b20ca6be30aa: Pull complete
6f8d52771bf7: Pull complete
9a075d558b4b: Pull complete
6ba6b39b87e7: Pull complete
fa5244c3e2fa: Pull complete
72edde940830: Pull complete
c618f323bf15: Pull complete
478e02023d40: Pull complete
d0e12a8666f5: Pull complete
9ba82d4ac0e8: Pull complete
83f21b096d92: Pull complete
7e847a87dea6: Pull complete
d0bd026e9517: Pull complete
f6b0b6c17afd: Pull complete
f5d244d620fb: Pull complete
3f16cf0cdc03: Pull complete
Digest: sha256:e19ff75603a5e74a82f39f897426119ad4c61cbbb5f0035b00c216d83f39190e
Status: Downloaded newer image for ghcr.io/apache/airflow/main/ci/python3.8:latest
Checking resources.
* Memory available 63G. OK.
* CPUs available 16. OK.
WARNING!!!: Not enough Disk space available for Docker.
At least 40 GBs recommended. You have 23G
WARNING!!!: You have not enough resources to run Airflow (see above)!
Please follow the instructions to increase amount of resources available:
Please check https://github.com/apache/airflow/blob/main/BREEZE.rst#resources-required for details
Good version of docker-compose: 1.29.2
WARNING: The ENABLE_TEST_COVERAGE variable is not set. Defaulting to a blank string.
Pulling mysql (mysql:5.7)...
5.7: Pulling from library/mysql
a10c77af2613: Already exists
b76a7eb51ffd: Pull complete
258223f927e4: Pull complete
2d2c75386df9: Pull complete
63e92e4046c9: Pull complete
f5845c731544: Pull complete
bd0401123a9b: Pull complete
2724b2da64fd: Pull complete
d10a7e9e325c: Pull complete
1c5fd9c3683d: Pull complete
2e35f83a12e9: Pull complete
Digest: sha256:7a3a7b7a29e6fbff433c339fc52245435fa2c308586481f2f92ab1df239d6a29
Status: Downloaded newer image for mysql:5.7
Creating docker-compose_mysql_1 ... done
Creating docker-compose_airflow_run ... done
Error response from daemon: driver failed programming external connectivity on endpoint docker-compose_airflow_run_f081fd6ac899 (0d04b319ee1c4e4ceaed9d626bc70df0cb1ae0699efdf9ccead83aaed72ca420): Bind for 0.0.0.0:28080 failed: port is already allocated
ERROR: 1
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org