You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/22 21:20:53 UTC
[GitHub] [airflow] j143 opened a new pull request #19756: gitpodify Apache Airflow - online development workspace
j143 opened a new pull request #19756:
URL: https://github.com/apache/airflow/pull/19756
At present, configuration startups up the ide
with `./breeze -y` for setting up breeze environment.
**How to test?**
1. Visiting the link [`apache/airflow/pull/16498`](https://gitpod.io/#https://github.com/apache/airflow/pull/16498) would fire up the online ready to code workspace.
2. There will be two terminals. In the left it is general, in the right you could run any tests with `breeze`
**Terminals:**
**Testing:**
<details>
<summary><code> pytest tests/core/test_core.py::TestCore::test_check_operators</code></summary>
```console
root@d143c0ff1e51:/opt/airflow# pytest tests/core/test_core.py::TestCore::test_check_operators
================================= test session starts ==================================
platform linux -- Python 3.6.13, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow, configfile: pytest.ini
plugins: httpx-0.12.0, cov-2.12.0, requests-mock-1.9.3, celery-4.4.7, forked-1.3.0, instafail-0.4.2, xdist-2.2.1, rerunfailures-9.1.1, flaky-3.7.0, timeouts-1.2.1
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 1 item
tests/core/test_core.py::TestCore::test_check_operators
PASSED [100%]
=================================== warnings summary ===================================
tests/core/test_core.py::TestCore::test_check_operators
/usr/local/lib/python3.6/importlib/__init__.py:126: DeprecationWarning: This module is deprecated. Please use `airflow.providers.tableau.hooks.tableau`.
return _bootstrap._gcd_import(name[level:], package, level)
tests/core/test_core.py::TestCore::test_check_operators
/usr/local/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
tests/core/test_core.py::TestCore::test_check_operators
/usr/local/lib/python3.6/site-packages/boto/plugin.py:40: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
tests/core/test_core.py::TestCore::test_check_operators
/usr/local/lib/python3.6/site-packages/dns/hash.py:25: DeprecationWarning: dns.hash module will be removed in future versions. Please use hashlib instead.
DeprecationWarning)
tests/core/test_core.py::TestCore::test_check_operators
/usr/local/lib/python3.6/site-packages/eventlet/green/OpenSSL/__init__.py:6: DeprecationWarning: OpenSSL.tsafe is deprecated and will be removed
from . import tsafe
tests/core/test_core.py::TestCore::test_check_operators
/usr/local/lib/python3.6/site-packages/alembic/ddl/sqlite.py:44: UserWarning: Skipping unsupported ALTER for creation of implicit constraintPlease refer to the batch mode feature which allows for SQLite migrations using a copy-and-move strategy.
"Skipping unsupported ALTER for "
tests/core/test_core.py::TestCore::test_check_operators
tests/core/test_core.py::TestCore::test_check_operators
tests/core/test_core.py::TestCore::test_check_operators
/usr/local/lib/python3.6/site-packages/flask_caching/__init__.py:241: DeprecationWarning: Using the initialization functions in flask_caching.backend is deprecated. Use the a full path to backend classes directly.
category=DeprecationWarning,
tests/core/test_core.py::TestCore::test_check_operators
tests/core/test_core.py::TestCore::test_check_operators
tests/core/test_core.py::TestCore::test_check_operators
tests/core/test_core.py::TestCore::test_check_operators
/usr/local/lib/python3.6/site-packages/marshmallow/fields.py:201: RemovedInMarshmallow4Warning: Passing field metadata as a keyword arg is deprecated. Use the explicit `metadata=...` argument instead.
RemovedInMarshmallow4Warning,
tests/core/test_core.py::TestCore::test_check_operators
/opt/airflow/tests/core/test_core.py:102: DeprecationWarning: This class is deprecated.
Please use `airflow.operators.sql.SQLCheckOperator`.
task_id='check', sql="select count(*) from operator_test_table", conn_id=conn_id, dag=self.dag
tests/core/test_core.py::TestCore::test_check_operators
/opt/airflow/tests/core/test_core.py:113: DeprecationWarning: This class is deprecated.
Please use `airflow.operators.sql.SQLValueCheckOperator`.
dag=self.dag,
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== 1 passed, 15 warnings in 37.48s ============================
```
</details>
**Problems encountered:**
1. "Error response from daemon: driver failed programming external connectivity on endpoint" while running `./breeze --integration mongo`
2. Which ports should be open to public/private ? (suggestions please.)
Related: #16480
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-982519849
@j143 - you might be interested that we **just** start the project of rewriting `Breeze` to python based version - first commit that implements the scaffolding and explain some decisions behind the project is here https://github.com/apache/airflow/pull/19867
You might want to contribute to it and eventually switch the gitpodified experience to it (and any comments/suggestions/improvemetns or contribution while we develop it is most welcome).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-975935701
copy to @potiuk and David Brownkush
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-982519849
@j43 - you might be interested that we **just** start the project of rewriting `Breeze` to python based version - first commit that implements the scaffolding and explain some decisions behind the project is here https://github.com/apache/airflow/pull/19867
You might want to contribute to it and eventually switch the gitpodified experience to it (and any comments/suggestions/improvemetns or contribution while we develop it is most welcome).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-976369440
Nice and Simple! :) This is what Breeze was created for :) . My initial goal was to get starrted with airflow under 10 minutes, so 5 minutes is pretty damn good.
Re: ports: I think all the ports that Breeze has comments about:
```
Ports are forwarded to the running docker containers for webserver and database
* 12322 -> forwarded to Airflow ssh server -> airflow:22
* 28080 -> forwarded to Airflow webserver -> airflow:8080
* 25555 -> forwarded to Flower dashboard -> airflow:5555
* 25433 -> forwarded to Postgres database -> postgres:5432
* 23306 -> forwarded to MySQL database -> mysql:3306
* 21433 -> forwarded to MSSQL database -> mssql:1443
* 26379 -> forwarded to Redis broker -> redis:6379
Here are links to those services that you can use on host:
* ssh connection for remote debugging: ssh -p 12322 airflow@127.0.0.1 pw: airflow
* Webserver: http://127.0.0.1:28080
* Flower: http://127.0.0.1:25555
* Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
* Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
* Redis: redis://127.0.0.1:26379/0
```
Re: mongo - not sure why the problems. there are some problems with docker-compose2 for integrations (and networking) so maybe worth checking if we can configure `docker-compose` version used.
Questions @j143 :
I do not know gitpod that much, but is there a way we could configure some "options" when starting such vm? for example it would be great if when starting the vm you could choose:
* backend
* python version
* integrations enabled
For integrations - maybe just some predefined sets of those would be enough: (`--integrations all` switch)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-976378199
Two more things:
a) licence is missing in the yml file
b) we should also install pre-commit automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756859620
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1818,3 +1818,192 @@ describes how to do it.
.. raw:: html
</details>
+
+
+Setup and develop using Gitpod online workspaces
+################################################
+
+.. raw:: html
+
+ <details>
+ <summary>Setup and develop using Gitpod online workspaces</summary>
+
+
+
+Setup Airflow with Breeze
+-------------------------
+
+
+Forking and cloning Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+3. Add goto https://gitpod.io/#<copied-url> as shown.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_gitpod_url.png"
+ alt="Open personal airflow clone with Gitpod">
+ </div>
+
+Setting up Breeze
+~~~~~~~~~~~~~~~~~
+
+1. Breeze is already initialized in one of the terminals in Gitpod
+
+2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
+ is required to execute at least once for Airflow Breeze to get the database/tables created.
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow# airflow db reset
+ root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
Review comment:
This should stay here - even if you do `--db-reset` but since it is only needed when you run/use webserver, I think you can specify that you need it only when you do.
I use airflow webserver extremely rarely when developing Airflow, and while it is useful to have it, it's mostly not needed to add core feature or provider.
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1818,3 +1818,192 @@ describes how to do it.
.. raw:: html
</details>
+
+
+Setup and develop using Gitpod online workspaces
+################################################
+
+.. raw:: html
+
+ <details>
+ <summary>Setup and develop using Gitpod online workspaces</summary>
+
+
+
+Setup Airflow with Breeze
+-------------------------
+
+
+Forking and cloning Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+3. Add goto https://gitpod.io/#<copied-url> as shown.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_gitpod_url.png"
+ alt="Open personal airflow clone with Gitpod">
+ </div>
+
+Setting up Breeze
+~~~~~~~~~~~~~~~~~
+
+1. Breeze is already initialized in one of the terminals in Gitpod
+
+2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
+ is required to execute at least once for Airflow Breeze to get the database/tables created.
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow# airflow db reset
+ root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
Review comment:
This should stay here - even if you do `--db-reset` but since it is only needed when you run/use webserver, I think you can specify that you need it only when you do.
I, for example, use airflow webserver extremely rarely when developing Airflow, and while it is useful to have it, it's mostly not needed to add core feature or provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-976375218
If all else fails - i think it would be possible with env variables. Breeze already supports reacting to the environment variables so you could pass them for your gp instance https://www.gitpod.io/docs/environment-variables.
Those will be:
* `BACKEND`
* `PYTHON_MAJOR_MINOR_VERSION`
* `INTEGRATIONS`
The last one is the list of integrations enabled.
I think for this one to be merged we need a separate "quick-start" - short version on how to start and how to configure the env variables) in https://github.com/apache/airflow/blob/main/CONTRIBUTORS_QUICK_START.rst .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756857755
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1818,3 +1818,192 @@ describes how to do it.
.. raw:: html
</details>
+
+
+Setup and develop using Gitpod online workspaces
+################################################
+
+.. raw:: html
+
+ <details>
+ <summary>Setup and develop using Gitpod online workspaces</summary>
+
+
+
+Setup Airflow with Breeze
+-------------------------
+
+
+Forking and cloning Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+3. Add goto https://gitpod.io/#<copied-url> as shown.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_gitpod_url.png"
+ alt="Open personal airflow clone with Gitpod">
+ </div>
+
+Setting up Breeze
+~~~~~~~~~~~~~~~~~
+
+1. Breeze is already initialized in one of the terminals in Gitpod
+
+2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
Review comment:
This might be gone if you use `-db-reset` switch when starting Breeze (see the other comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756848700
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1896,3 +1896,114 @@ Setting up Breeze
$ ./breeze stop
+Installing airflow with breeze.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Gitpod default image have all the required packages installed.
+
+1. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/Projects/airflow"
+ source ~/.bashrc
+
+
+Starting development
+--------------------
+
+
+Creating a branch
+~~~~~~~~~~~~~~~~~
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_2.png"
+ alt="Giving a name to a branch">
+ </div>
+
+
+
+Testing
+~~~~~~~
+
+All Tests are inside ./tests directory.
Review comment:
```suggestion
All Tests are inside ``./tests`` directory.
```
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1896,3 +1896,114 @@ Setting up Breeze
$ ./breeze stop
+Installing airflow with breeze.
Review comment:
```suggestion
Installing Airflow with Breeze.
```
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1896,3 +1896,114 @@ Setting up Breeze
$ ./breeze stop
+Installing airflow with breeze.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Gitpod default image have all the required packages installed.
+
+1. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/Projects/airflow"
+ source ~/.bashrc
+
+
+Starting development
+--------------------
+
+
+Creating a branch
+~~~~~~~~~~~~~~~~~
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_2.png"
+ alt="Giving a name to a branch">
+ </div>
+
+
+
+Testing
+~~~~~~~
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# pytest tests/utils/test_session.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, cov-3.0.0, forked-1.3.0, httpx-0.15.0, instafail-0.4.2, rerunfailures-9.1.1, timeouts-1.2.1, xdist-2.4.0, requests-mock-1.9.3
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 4 items
+
+ tests/utils/test_session.py::TestSession::test_raised_provide_session PASSED [ 25%]
+ tests/utils/test_session.py::TestSession::test_provide_session_without_args_and_kwargs PASSED [ 50%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_args PASSED [ 75%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_kwargs PASSED [100%]
+
+====================================== 4 passed, 11 warnings in 33.14s ======================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
Review comment:
```suggestion
- Running All the test with Breeze by specifying required Python version, backend, backend version
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 edited a comment on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 edited a comment on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979299281
> How about the comments with `--db-reset` ?
I thought, I have added a comment on breeze stop.
I have added this notes as far I understood - https://github.com/apache/airflow/pull/19756/commits/fc45c3e3cfa7e442b85f66573f547888791882bd
please provide any comments on that. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979218131
Hi Jarek,
I have added the basic docs, and remaining tasks I have added as a checklist at #16480
Thanks for review and suggestion.
> fun thing: I have done all the development, on the web browser itself. I pressed `.` once on my keyboard!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19756:
URL: https://github.com/apache/airflow/pull/19756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-982509456
Hi @potiuk , I have rebased it recently into one single commit. I hope main points were addressed. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-976383139
One more cool thing while we are adding it, what's interesting is this one: https://www.gitpod.io/docs/environment-variables#provide-env-vars-via-url.
It should be a follow-up PR but tt would be great if we can add an option to replicate CI failed builds in GitPod environment - seems with this one it should be possible. So it should be essentially possibleo add instructions for the user on how to replicate CI failed build in their gitpod environment.
Look here:
https://github.com/apache/airflow/blob/main/scripts/ci/testing/ci_run_single_airflow_test_in_docker.sh#L162
Essentially we should be able to craft an URL that should create a gitpodify environment for this specific build configuration:
1) It will pull the right image from the CI build
2) it will set the right environment (Backend. python version, integrations etc).
3) you will be able toreplicate failed tests directly there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-976383139
One more cool thing while we are adding it, what's interesting is this one: https://www.gitpod.io/docs/environment-variables#provide-env-vars-via-url.
It should be a follow-up PR but tt would be great if we can add an option to replicate CI failed builds in GitPod environment - seems with this one it should be possible. So it should be essentially possible (next PR) to add an option for the user on how to replicate CI failed build in their gitpod environment.
Look here:
https://github.com/apache/airflow/blob/main/scripts/ci/testing/ci_run_single_airflow_test_in_docker.sh#L162
Essentially we should be able to craft an URL that should create a gitpodify environment for this specific build configuration:
1) It will pull the right image from the CI build
2) it will set the right environment (Backend. python version, integrations etc).
3) you will be able toreplicate failed tests directly there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-976385987
I really like how simple it is to make the environment works with GitPod + Breeze :). We'll do very similar thing for Codespaces when they are publicly available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r754963311
##########
File path: .gitpod.yml
##########
@@ -0,0 +1,12 @@
+# Reference: https://www.gitpod.io/docs/references/gitpod-yml
+# Docs: https://www.gitpod.io/docs/config-gitpod-file/
+
+tasks:
+ - init: ./breeze -y
+
+ - openMode: split-right
+ command: echo for running integration test with breeze
+# Ports to expose on workspace startup
+ports:
+ - port: 8000
+ onOpen: open-preview
Review comment:
```suggestion
onOpen: open-preview
```
Need that final line break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756856362
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1818,3 +1818,192 @@ describes how to do it.
.. raw:: html
</details>
+
+
+Setup and develop using Gitpod online workspaces
+################################################
+
+.. raw:: html
+
+ <details>
+ <summary>Setup and develop using Gitpod online workspaces</summary>
+
+
+
+Setup Airflow with Breeze
+-------------------------
+
+
+Forking and cloning Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+3. Add goto https://gitpod.io/#<copied-url> as shown.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_gitpod_url.png"
+ alt="Open personal airflow clone with Gitpod">
+ </div>
+
+Setting up Breeze
+~~~~~~~~~~~~~~~~~
+
+1. Breeze is already initialized in one of the terminals in Gitpod
+
+2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
+ is required to execute at least once for Airflow Breeze to get the database/tables created.
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow# airflow db reset
+ root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
+ --email admin@example.com --firstname foo --lastname bar
+
+
+3. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+
+Installing airflow with breeze.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Gitpod default image have all the required packages installed.
+
+1. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/Projects/airflow"
+ source ~/.bashrc
+
+
+Starting development
+--------------------
+
+
+Creating a branch
+~~~~~~~~~~~~~~~~~
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_2.png"
+ alt="Giving a name to a branch">
+ </div>
+
+
+
+Testing
+~~~~~~~
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# pytest tests/utils/test_session.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, cov-3.0.0, forked-1.3.0, httpx-0.15.0, instafail-0.4.2, rerunfailures-9.1.1, timeouts-1.2.1, xdist-2.4.0, requests-mock-1.9.3
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 4 items
+
+ tests/utils/test_session.py::TestSession::test_raised_provide_session PASSED [ 25%]
+ tests/utils/test_session.py::TestSession::test_provide_session_without_args_and_kwargs PASSED [ 50%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_args PASSED [ 75%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_kwargs PASSED [100%]
+
+====================================== 4 passed, 11 warnings in 33.14s ======================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``./scripts/in_container`` directory.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# ls ./scripts/in_container/
+ _in_container_script_init.sh quarantine_issue_header.md run_mypy.sh
+ _in_container_utils.sh run_anything.sh run_prepare_airflow_packages.sh
+ airflow_ci.cfg run_ci_tests.sh run_prepare_provider_documentation.sh
+ bin run_docs_build.sh run_prepare_provider_packages.sh
+ check_environment.sh run_extract_tests.sh run_resource_check.sh
+ check_junitxml_result.py run_fix_ownership.sh run_system_tests.sh
+ configure_environment.sh run_flake8.sh run_tmux_welcome.sh
+ entrypoint_ci.sh run_generate_constraints.sh stop_tmux_airflow.sh
+ entrypoint_exec.sh run_init_script.sh update_quarantined_test_status.py
+ prod run_install_and_test_provider_packages.sh
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+- Running specific type of test
+
+ - Types of tests
+
+ - Running specific type of test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Review comment:
Yeah - fixed ports unfortunately :(. I think it might be a good idea to run `breeze stop` beore starting a new instance. It will also make sure that all the DB volumes are cleared and databased will be "fresh like a daisy". You can also actually start `breeze` always with `--db-reset` switch - this will make sure that every time you initialize gitpod environment the database will be recreated. This is a nice feature - especially if you plan switch back/forth between branches and the environment will be preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979224904
BTW. Static checks are failing :). That's why it would have been great to integrate pre-commit from get-go :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-976375218
If all else fails - i think it would be possible with env variables. Breeze already supports reacting to the environment variables so you could pass them for your gp instance https://www.gitpod.io/docs/environment-variables.
Those will be:
BACKEND
PYTHON_MAJOR_MINOR_VERSIONS
INTEGRATIONS
The last one is the list of integrations enabled.
I think for this one to be merged we need a separate "quick-start" - short version on how to start and how to configure the env variables) in https://github.com/apache/airflow/blob/main/CONTRIBUTORS_QUICK_START.rst .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756838115
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1818,3 +1818,192 @@ describes how to do it.
.. raw:: html
</details>
+
+
+Setup and develop using Gitpod online workspaces
+################################################
+
+.. raw:: html
+
+ <details>
+ <summary>Setup and develop using Gitpod online workspaces</summary>
+
+
+
+Setup Airflow with Breeze
+-------------------------
+
+
+Forking and cloning Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+3. Add goto https://gitpod.io/#<copied-url> as shown.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_gitpod_url.png"
+ alt="Open personal airflow clone with Gitpod">
+ </div>
+
+Setting up Breeze
+~~~~~~~~~~~~~~~~~
+
+1. Breeze is already initialized in one of the terminals in Gitpod
+
+2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
+ is required to execute at least once for Airflow Breeze to get the database/tables created.
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow# airflow db reset
+ root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
+ --email admin@example.com --firstname foo --lastname bar
+
+
+3. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+
+Installing airflow with breeze.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Gitpod default image have all the required packages installed.
+
+1. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/Projects/airflow"
+ source ~/.bashrc
+
+
+Starting development
+--------------------
+
+
+Creating a branch
+~~~~~~~~~~~~~~~~~
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_2.png"
+ alt="Giving a name to a branch">
+ </div>
+
+
+
+Testing
+~~~~~~~
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# pytest tests/utils/test_session.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, cov-3.0.0, forked-1.3.0, httpx-0.15.0, instafail-0.4.2, rerunfailures-9.1.1, timeouts-1.2.1, xdist-2.4.0, requests-mock-1.9.3
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 4 items
+
+ tests/utils/test_session.py::TestSession::test_raised_provide_session PASSED [ 25%]
+ tests/utils/test_session.py::TestSession::test_provide_session_without_args_and_kwargs PASSED [ 50%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_args PASSED [ 75%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_kwargs PASSED [100%]
+
+====================================== 4 passed, 11 warnings in 33.14s ======================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``./scripts/in_container`` directory.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# ls ./scripts/in_container/
+ _in_container_script_init.sh quarantine_issue_header.md run_mypy.sh
+ _in_container_utils.sh run_anything.sh run_prepare_airflow_packages.sh
+ airflow_ci.cfg run_ci_tests.sh run_prepare_provider_documentation.sh
+ bin run_docs_build.sh run_prepare_provider_packages.sh
+ check_environment.sh run_extract_tests.sh run_resource_check.sh
+ check_junitxml_result.py run_fix_ownership.sh run_system_tests.sh
+ configure_environment.sh run_flake8.sh run_tmux_welcome.sh
+ entrypoint_ci.sh run_generate_constraints.sh stop_tmux_airflow.sh
+ entrypoint_exec.sh run_init_script.sh update_quarantined_test_status.py
+ prod run_install_and_test_provider_packages.sh
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+- Running specific type of test
+
+ - Types of tests
+
+ - Running specific type of test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Review comment:
This command gives an error for me.
<details>
<summary>error log</summary>
```c
gitpod /workspace/airflow $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Good version of docker 20.10.8.
Backend: mysql
MySQL version: 5.7
Python version: 3.8
Resetting the DB!
Selected test type: Core
mkdir: created directory '/workspace/airflow/.build/main/3.8'
mkdir: created directory '/workspace/airflow/.build/main/3.8/CI'
a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362
a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362
Use CI image.
Branch name: main
Docker image: ghcr.io/apache/airflow/main/ci/python3.8:latest
Airflow source version: 2.3.0.dev0
Python version: 3.8
Backend: mysql 5.7
####################################################################################################
Airflow Breeze CHEATSHEET
/workspace/airflow/breeze
####################################################################################################
Port forwarding:
Ports are forwarded to the running docker containers for webserver and database
* 12322 -> forwarded to Airflow ssh server -> airflow:22
* 28080 -> forwarded to Airflow webserver -> airflow:8080
* 25555 -> forwarded to Flower dashboard -> airflow:5555
* 25433 -> forwarded to Postgres database -> postgres:5432
* 23306 -> forwarded to MySQL database -> mysql:3306
* 21433 -> forwarded to MSSQL database -> mssql:1443
* 26379 -> forwarded to Redis broker -> redis:6379
Here are links to those services that you can use on host:
* ssh connection for remote debugging: ssh -p 12322 airflow@127.0.0.1 pw: airflow
* Webserver: http://127.0.0.1:28080
* Flower: http://127.0.0.1:25555
* Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
* Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
* Redis: redis://127.0.0.1:26379/0
####################################################################################################
You can setup autocomplete by running 'breeze setup-autocomplete'
####################################################################################################
You can toggle ascii/cheatsheet by running:
* breeze toggle-suppress-cheatsheet
* breeze toggle-suppress-asciiart
####################################################################################################
Unable to find image 'ghcr.io/apache/airflow/main/ci/python3.8:latest' locally
ylatest: Pulling from apache/airflow/main/ci/python3.8
a10c77af2613: Already exists
811275ca69b8: Already exists
a6e65fb20e69: Pulling fs layer
fc5582d0981c: Pulling fs layer
9c7a8f0f8e58: Pulling fs layer
8441add4ab92: Pulling fs layer
ebda9c5a1db7: Pulling fs layer
dbad869b28d4: Pulling fs layer
2a66638f1e61: Pulling fs layer
41837b664714: Pulling fs layer
a771646b27d5: Pulling fs layer
afb4979590d9: Pulling fs layer
428b7c9025e2: Pulling fs layer
bb98aff13f15: Pulling fs layer
9cd5fdf5c4d5: Pulling fs layer
6899d37becd2: Pulling fs layer
58f7721b779c: Pulling fs layer
b20ca6be30aa: Pulling fs layer
8441add4ab92: Waiting
ebda9c5a1db7: Waiting
6f8d52771bf7: Pulling fs layer
9a075d558b4b: Pulling fs layer
428b7c9025e2: Waiting
6ba6b39b87e7: Pulling fs layer
fa5244c3e2fa: Pulling fs layer
72edde940830: Pulling fs layer
2a66638f1e61: Waiting
afb4979590d9: Waiting
41837b664714: Waiting
c618f323bf15: Pulling fs layer
6f8d52771bf7: Waiting
9cd5fdf5c4d5: Waiting
6ba6b39b87e7: Waiting
bb98aff13f15: Waiting
478e02023d40: Pulling fs layer
58f7721b779c: Waiting
d0e12a8666f5: Pulling fs layer
fa5244c3e2fa: Waiting
9ba82d4ac0e8: Pulling fs layer
c618f323bf15: Waiting
478e02023d40: Waiting
83f21b096d92: Pulling fs layer
b20ca6be30aa: Waiting
72edde940830: Waiting
d0e12a8666f5: Waiting
6899d37becd2: Waiting
7e847a87dea6: Pulling fs layer
d0bd026e9517: Pulling fs layer
9a075d558b4b: Waiting
f6b0b6c17afd: Pulling fs layer
7e847a87dea6: Waiting
f5d244d620fb: Pulling fs layer
d0bd026e9517: Waiting
f6b0b6c17afd: Waiting
3f16cf0cdc03: Pulling fs layer
f5d244d620fb: Waiting
fc5582d0981c: Verifying Checksum
fc5582d0981c: Download complete
9c7a8f0f8e58: Verifying Checksum
9c7a8f0f8e58: Download complete
a6e65fb20e69: Verifying Checksum
a6e65fb20e69: Download complete
8441add4ab92: Verifying Checksum
8441add4ab92: Download complete
dbad869b28d4: Verifying Checksum
dbad869b28d4: Download complete
41837b664714: Verifying Checksum
41837b664714: Download complete
2a66638f1e61: Verifying Checksum
2a66638f1e61: Download complete
a6e65fb20e69: Pull complete
fc5582d0981c: Pull complete
afb4979590d9: Verifying Checksum
afb4979590d9: Download complete
428b7c9025e2: Verifying Checksum
428b7c9025e2: Download complete
9c7a8f0f8e58: Pull complete
bb98aff13f15: Verifying Checksum
bb98aff13f15: Download complete
a771646b27d5: Verifying Checksum
a771646b27d5: Download complete
6899d37becd2: Verifying Checksum
6899d37becd2: Download complete
8441add4ab92: Pull complete
58f7721b779c: Verifying Checksum
58f7721b779c: Download complete
ebda9c5a1db7: Download complete
b20ca6be30aa: Download complete
9a075d558b4b: Verifying Checksum
9a075d558b4b: Download complete
6ba6b39b87e7: Download complete
fa5244c3e2fa: Verifying Checksum
fa5244c3e2fa: Download complete
6f8d52771bf7: Verifying Checksum
6f8d52771bf7: Download complete
72edde940830: Verifying Checksum
72edde940830: Download complete
c618f323bf15: Verifying Checksum
c618f323bf15: Download complete
478e02023d40: Verifying Checksum
478e02023d40: Download complete
d0e12a8666f5: Verifying Checksum
d0e12a8666f5: Download complete
83f21b096d92: Download complete
9ba82d4ac0e8: Verifying Checksum
9ba82d4ac0e8: Download complete
7e847a87dea6: Download complete
d0bd026e9517: Verifying Checksum
d0bd026e9517: Download complete
f6b0b6c17afd: Verifying Checksum
f6b0b6c17afd: Download complete
f5d244d620fb: Verifying Checksum
f5d244d620fb: Download complete
3f16cf0cdc03: Verifying Checksum
3f16cf0cdc03: Download complete
9cd5fdf5c4d5: Verifying Checksum
9cd5fdf5c4d5: Download complete
ebda9c5a1db7: Pull complete
dbad869b28d4: Pull complete
2a66638f1e61: Pull complete
41837b664714: Pull complete
a771646b27d5: Pull complete
afb4979590d9: Pull complete
428b7c9025e2: Pull complete
bb98aff13f15: Pull complete
9cd5fdf5c4d5: Pull complete
6899d37becd2: Pull complete
58f7721b779c: Pull complete
b20ca6be30aa: Pull complete
6f8d52771bf7: Pull complete
9a075d558b4b: Pull complete
6ba6b39b87e7: Pull complete
fa5244c3e2fa: Pull complete
72edde940830: Pull complete
c618f323bf15: Pull complete
478e02023d40: Pull complete
d0e12a8666f5: Pull complete
9ba82d4ac0e8: Pull complete
83f21b096d92: Pull complete
7e847a87dea6: Pull complete
d0bd026e9517: Pull complete
f6b0b6c17afd: Pull complete
f5d244d620fb: Pull complete
3f16cf0cdc03: Pull complete
Digest: sha256:e19ff75603a5e74a82f39f897426119ad4c61cbbb5f0035b00c216d83f39190e
Status: Downloaded newer image for ghcr.io/apache/airflow/main/ci/python3.8:latest
Checking resources.
* Memory available 63G. OK.
* CPUs available 16. OK.
WARNING!!!: Not enough Disk space available for Docker.
At least 40 GBs recommended. You have 23G
WARNING!!!: You have not enough resources to run Airflow (see above)!
Please follow the instructions to increase amount of resources available:
Please check https://github.com/apache/airflow/blob/main/BREEZE.rst#resources-required for details
Good version of docker-compose: 1.29.2
WARNING: The ENABLE_TEST_COVERAGE variable is not set. Defaulting to a blank string.
Pulling mysql (mysql:5.7)...
5.7: Pulling from library/mysql
a10c77af2613: Already exists
b76a7eb51ffd: Pull complete
258223f927e4: Pull complete
2d2c75386df9: Pull complete
63e92e4046c9: Pull complete
f5845c731544: Pull complete
bd0401123a9b: Pull complete
2724b2da64fd: Pull complete
d10a7e9e325c: Pull complete
1c5fd9c3683d: Pull complete
2e35f83a12e9: Pull complete
Digest: sha256:7a3a7b7a29e6fbff433c339fc52245435fa2c308586481f2f92ab1df239d6a29
Status: Downloaded newer image for mysql:5.7
Creating docker-compose_mysql_1 ... done
Creating docker-compose_airflow_run ... done
Error response from daemon: driver failed programming external connectivity on endpoint docker-compose_airflow_run_f081fd6ac899 (0d04b319ee1c4e4ceaed9d626bc70df0cb1ae0699efdf9ccead83aaed72ca420): Bind for 0.0.0.0:28080 failed: port is already allocated
ERROR: 1
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756838115
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1818,3 +1818,192 @@ describes how to do it.
.. raw:: html
</details>
+
+
+Setup and develop using Gitpod online workspaces
+################################################
+
+.. raw:: html
+
+ <details>
+ <summary>Setup and develop using Gitpod online workspaces</summary>
+
+
+
+Setup Airflow with Breeze
+-------------------------
+
+
+Forking and cloning Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+3. Add goto https://gitpod.io/#<copied-url> as shown.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_gitpod_url.png"
+ alt="Open personal airflow clone with Gitpod">
+ </div>
+
+Setting up Breeze
+~~~~~~~~~~~~~~~~~
+
+1. Breeze is already initialized in one of the terminals in Gitpod
+
+2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
+ is required to execute at least once for Airflow Breeze to get the database/tables created.
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow# airflow db reset
+ root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
+ --email admin@example.com --firstname foo --lastname bar
+
+
+3. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+
+Installing airflow with breeze.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Gitpod default image have all the required packages installed.
+
+1. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/Projects/airflow"
+ source ~/.bashrc
+
+
+Starting development
+--------------------
+
+
+Creating a branch
+~~~~~~~~~~~~~~~~~
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_2.png"
+ alt="Giving a name to a branch">
+ </div>
+
+
+
+Testing
+~~~~~~~
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# pytest tests/utils/test_session.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, cov-3.0.0, forked-1.3.0, httpx-0.15.0, instafail-0.4.2, rerunfailures-9.1.1, timeouts-1.2.1, xdist-2.4.0, requests-mock-1.9.3
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 4 items
+
+ tests/utils/test_session.py::TestSession::test_raised_provide_session PASSED [ 25%]
+ tests/utils/test_session.py::TestSession::test_provide_session_without_args_and_kwargs PASSED [ 50%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_args PASSED [ 75%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_kwargs PASSED [100%]
+
+====================================== 4 passed, 11 warnings in 33.14s ======================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``./scripts/in_container`` directory.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# ls ./scripts/in_container/
+ _in_container_script_init.sh quarantine_issue_header.md run_mypy.sh
+ _in_container_utils.sh run_anything.sh run_prepare_airflow_packages.sh
+ airflow_ci.cfg run_ci_tests.sh run_prepare_provider_documentation.sh
+ bin run_docs_build.sh run_prepare_provider_packages.sh
+ check_environment.sh run_extract_tests.sh run_resource_check.sh
+ check_junitxml_result.py run_fix_ownership.sh run_system_tests.sh
+ configure_environment.sh run_flake8.sh run_tmux_welcome.sh
+ entrypoint_ci.sh run_generate_constraints.sh stop_tmux_airflow.sh
+ entrypoint_exec.sh run_init_script.sh update_quarantined_test_status.py
+ prod run_install_and_test_provider_packages.sh
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+- Running specific type of test
+
+ - Types of tests
+
+ - Running specific type of test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Review comment:
This command gives an error for me. that the port `28080` is already allocated, I also have the container running (may be that is the cause!).
<details>
<summary>error log</summary>
```c
gitpod /workspace/airflow $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Good version of docker 20.10.8.
Backend: mysql
MySQL version: 5.7
Python version: 3.8
Resetting the DB!
Selected test type: Core
mkdir: created directory '/workspace/airflow/.build/main/3.8'
mkdir: created directory '/workspace/airflow/.build/main/3.8/CI'
a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362
a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362
Use CI image.
Branch name: main
Docker image: ghcr.io/apache/airflow/main/ci/python3.8:latest
Airflow source version: 2.3.0.dev0
Python version: 3.8
Backend: mysql 5.7
####################################################################################################
Airflow Breeze CHEATSHEET
/workspace/airflow/breeze
####################################################################################################
Port forwarding:
Ports are forwarded to the running docker containers for webserver and database
* 12322 -> forwarded to Airflow ssh server -> airflow:22
* 28080 -> forwarded to Airflow webserver -> airflow:8080
* 25555 -> forwarded to Flower dashboard -> airflow:5555
* 25433 -> forwarded to Postgres database -> postgres:5432
* 23306 -> forwarded to MySQL database -> mysql:3306
* 21433 -> forwarded to MSSQL database -> mssql:1443
* 26379 -> forwarded to Redis broker -> redis:6379
Here are links to those services that you can use on host:
* ssh connection for remote debugging: ssh -p 12322 airflow@127.0.0.1 pw: airflow
* Webserver: http://127.0.0.1:28080
* Flower: http://127.0.0.1:25555
* Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
* Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
* Redis: redis://127.0.0.1:26379/0
####################################################################################################
You can setup autocomplete by running 'breeze setup-autocomplete'
####################################################################################################
You can toggle ascii/cheatsheet by running:
* breeze toggle-suppress-cheatsheet
* breeze toggle-suppress-asciiart
####################################################################################################
Unable to find image 'ghcr.io/apache/airflow/main/ci/python3.8:latest' locally
ylatest: Pulling from apache/airflow/main/ci/python3.8
a10c77af2613: Already exists
...
Digest: sha256:e19ff75603a5e74a82f39f897426119ad4c61cbbb5f0035b00c216d83f39190e
Status: Downloaded newer image for ghcr.io/apache/airflow/main/ci/python3.8:latest
Checking resources.
* Memory available 63G. OK.
* CPUs available 16. OK.
WARNING!!!: Not enough Disk space available for Docker.
At least 40 GBs recommended. You have 23G
WARNING!!!: You have not enough resources to run Airflow (see above)!
Please follow the instructions to increase amount of resources available:
Please check https://github.com/apache/airflow/blob/main/BREEZE.rst#resources-required for details
Good version of docker-compose: 1.29.2
WARNING: The ENABLE_TEST_COVERAGE variable is not set. Defaulting to a blank string.
Pulling mysql (mysql:5.7)...
5.7: Pulling from library/mysql
2e35f83a12e9: Pull complete
Digest: sha256:7a3a7b7a29e6fbff433c339fc52245435fa2c308586481f2f92ab1df239d6a29
Status: Downloaded newer image for mysql:5.7
Creating docker-compose_mysql_1 ... done
Creating docker-compose_airflow_run ... done
Error response from daemon: driver failed programming external connectivity on endpoint docker-compose_airflow_run_f081fd6ac899 (0d04b319ee1c4e4ceaed9d626bc70df0cb1ae0699efdf9ccead83aaed72ca420): Bind for 0.0.0.0:28080 failed: port is already allocated
ERROR: 1
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756885751
##########
File path: .gitpod.yml
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
Review comment:
```suggestion
---
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979222909
> > fun thing: I have done all the development, on the web browser itself. I pressed `.` once on my keyboard!
Oh yeah. That's ALMOST codespaces. Actually I already have access to codespaces and I want to do very same thing you did for gitpodify to make breeze starts when you enter codespaces :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979299281
> How about the comments with `--db-reset` ?
I thought, I have added a comment on breeze stop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on a change in pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on a change in pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#discussion_r756838115
##########
File path: CONTRIBUTORS_QUICK_START.rst
##########
@@ -1818,3 +1818,192 @@ describes how to do it.
.. raw:: html
</details>
+
+
+Setup and develop using Gitpod online workspaces
+################################################
+
+.. raw:: html
+
+ <details>
+ <summary>Setup and develop using Gitpod online workspaces</summary>
+
+
+
+Setup Airflow with Breeze
+-------------------------
+
+
+Forking and cloning Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ <a href="https://github.com/apache/airflow/" target="_blank">https://github.com/apache/airflow/</a>
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_fork.png"
+ alt="Forking Apache Airflow project">
+ </div>
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_clone.png"
+ alt="Cloning github fork of Apache airflow">
+ </div>
+
+3. Add goto https://gitpod.io/#<copied-url> as shown.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/airflow_gitpod_url.png"
+ alt="Open personal airflow clone with Gitpod">
+ </div>
+
+Setting up Breeze
+~~~~~~~~~~~~~~~~~
+
+1. Breeze is already initialized in one of the terminals in Gitpod
+
+2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
+ is required to execute at least once for Airflow Breeze to get the database/tables created.
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow# airflow db reset
+ root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
+ --email admin@example.com --firstname foo --lastname bar
+
+
+3. Closing Breeze environment. After successfully finishing above command will leave you in container,
+ type ``exit`` to exit the container
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow#
+ root@b76fcb399bb6:/opt/airflow# exit
+
+.. code-block:: bash
+
+ $ ./breeze stop
+
+
+Installing airflow with breeze.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Gitpod default image have all the required packages installed.
+
+1. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+
+ export PATH=${PATH}:"/home/${USER}/Projects/airflow"
+ source ~/.bashrc
+
+
+Starting development
+--------------------
+
+
+Creating a branch
+~~~~~~~~~~~~~~~~~
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_1.png"
+ alt="Creating a new branch">
+ </div>
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/quick_start/vscode_creating_branch_2.png"
+ alt="Giving a name to a branch">
+ </div>
+
+
+
+Testing
+~~~~~~~
+
+All Tests are inside ./tests directory.
+
+- Running Unit tests inside Breeze environment.
+
+ Just run ``pytest filepath+filename`` to run the tests.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# pytest tests/utils/test_session.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, cov-3.0.0, forked-1.3.0, httpx-0.15.0, instafail-0.4.2, rerunfailures-9.1.1, timeouts-1.2.1, xdist-2.4.0, requests-mock-1.9.3
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 4 items
+
+ tests/utils/test_session.py::TestSession::test_raised_provide_session PASSED [ 25%]
+ tests/utils/test_session.py::TestSession::test_provide_session_without_args_and_kwargs PASSED [ 50%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_args PASSED [ 75%]
+ tests/utils/test_session.py::TestSession::test_provide_session_with_kwargs PASSED [100%]
+
+====================================== 4 passed, 11 warnings in 33.14s ======================================
+
+- Running All the test with Breeze by specifying required python version, backend, backend version
+
+.. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
+
+
+- Running specific test in container using shell scripts. Testing in container scripts are located in
+ ``./scripts/in_container`` directory.
+
+.. code-block:: bash
+
+ root@4a2143c17426:/opt/airflow# ls ./scripts/in_container/
+ _in_container_script_init.sh quarantine_issue_header.md run_mypy.sh
+ _in_container_utils.sh run_anything.sh run_prepare_airflow_packages.sh
+ airflow_ci.cfg run_ci_tests.sh run_prepare_provider_documentation.sh
+ bin run_docs_build.sh run_prepare_provider_packages.sh
+ check_environment.sh run_extract_tests.sh run_resource_check.sh
+ check_junitxml_result.py run_fix_ownership.sh run_system_tests.sh
+ configure_environment.sh run_flake8.sh run_tmux_welcome.sh
+ entrypoint_ci.sh run_generate_constraints.sh stop_tmux_airflow.sh
+ entrypoint_exec.sh run_init_script.sh update_quarantined_test_status.py
+ prod run_install_and_test_provider_packages.sh
+
+ root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
+
+- Running specific type of test
+
+ - Types of tests
+
+ - Running specific type of test
+
+ .. code-block:: bash
+
+ $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Review comment:
This command gives an error for me.
<details>
<summary>error log</summary>
```c
gitpod /workspace/airflow $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
Good version of docker 20.10.8.
Backend: mysql
MySQL version: 5.7
Python version: 3.8
Resetting the DB!
Selected test type: Core
mkdir: created directory '/workspace/airflow/.build/main/3.8'
mkdir: created directory '/workspace/airflow/.build/main/3.8/CI'
a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362
a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362
Use CI image.
Branch name: main
Docker image: ghcr.io/apache/airflow/main/ci/python3.8:latest
Airflow source version: 2.3.0.dev0
Python version: 3.8
Backend: mysql 5.7
####################################################################################################
Airflow Breeze CHEATSHEET
/workspace/airflow/breeze
####################################################################################################
Port forwarding:
Ports are forwarded to the running docker containers for webserver and database
* 12322 -> forwarded to Airflow ssh server -> airflow:22
* 28080 -> forwarded to Airflow webserver -> airflow:8080
* 25555 -> forwarded to Flower dashboard -> airflow:5555
* 25433 -> forwarded to Postgres database -> postgres:5432
* 23306 -> forwarded to MySQL database -> mysql:3306
* 21433 -> forwarded to MSSQL database -> mssql:1443
* 26379 -> forwarded to Redis broker -> redis:6379
Here are links to those services that you can use on host:
* ssh connection for remote debugging: ssh -p 12322 airflow@127.0.0.1 pw: airflow
* Webserver: http://127.0.0.1:28080
* Flower: http://127.0.0.1:25555
* Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
* Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
* Redis: redis://127.0.0.1:26379/0
####################################################################################################
You can setup autocomplete by running 'breeze setup-autocomplete'
####################################################################################################
You can toggle ascii/cheatsheet by running:
* breeze toggle-suppress-cheatsheet
* breeze toggle-suppress-asciiart
####################################################################################################
Unable to find image 'ghcr.io/apache/airflow/main/ci/python3.8:latest' locally
ylatest: Pulling from apache/airflow/main/ci/python3.8
a10c77af2613: Already exists
...
Digest: sha256:e19ff75603a5e74a82f39f897426119ad4c61cbbb5f0035b00c216d83f39190e
Status: Downloaded newer image for ghcr.io/apache/airflow/main/ci/python3.8:latest
Checking resources.
* Memory available 63G. OK.
* CPUs available 16. OK.
WARNING!!!: Not enough Disk space available for Docker.
At least 40 GBs recommended. You have 23G
WARNING!!!: You have not enough resources to run Airflow (see above)!
Please follow the instructions to increase amount of resources available:
Please check https://github.com/apache/airflow/blob/main/BREEZE.rst#resources-required for details
Good version of docker-compose: 1.29.2
WARNING: The ENABLE_TEST_COVERAGE variable is not set. Defaulting to a blank string.
Pulling mysql (mysql:5.7)...
5.7: Pulling from library/mysql
2e35f83a12e9: Pull complete
Digest: sha256:7a3a7b7a29e6fbff433c339fc52245435fa2c308586481f2f92ab1df239d6a29
Status: Downloaded newer image for mysql:5.7
Creating docker-compose_mysql_1 ... done
Creating docker-compose_airflow_run ... done
Error response from daemon: driver failed programming external connectivity on endpoint docker-compose_airflow_run_f081fd6ac899 (0d04b319ee1c4e4ceaed9d626bc70df0cb1ae0699efdf9ccead83aaed72ca420): Bind for 0.0.0.0:28080 failed: port is already allocated
ERROR: 1
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979329162
> please provide any comments on that. :)
I think maybe (not 100% sure if you think it's a good idea) you shoud add `--db-reset` in the command starting breeze in the gitpod configuration (`./breeze --db-reset -y`) . This way you could simply remove the instruction about "running airlfow db init once" - as the db will be initialized when you enter breeze. This has the drawback that every time you enter the environment again, the db will be cleaned, but I think it happens anyway - I do not know for how long the remote VMs are cached and whether you always recreate it when you reconnect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-976369440
Nice and Simple! :) This is what Breeze was created for :) . My initial goal was to get starrted with airflow under 10 minutes, so 5 minutes is pretty damn good.
Re: ports: I thin all the ports that Breeze has ccomments about:
```
Ports are forwarded to the running docker containers for webserver and database
* 12322 -> forwarded to Airflow ssh server -> airflow:22
* 28080 -> forwarded to Airflow webserver -> airflow:8080
* 25555 -> forwarded to Flower dashboard -> airflow:5555
* 25433 -> forwarded to Postgres database -> postgres:5432
* 23306 -> forwarded to MySQL database -> mysql:3306
* 21433 -> forwarded to MSSQL database -> mssql:1443
* 26379 -> forwarded to Redis broker -> redis:6379
Here are links to those services that you can use on host:
* ssh connection for remote debugging: ssh -p 12322 airflow@127.0.0.1 pw: airflow
* Webserver: http://127.0.0.1:28080
* Flower: http://127.0.0.1:25555
* Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
* Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
* Redis: redis://127.0.0.1:26379/0
```
Re: mongo - not sure why the problems. there are some problems with docker-compose2 for integrations (and networking) so maybe worth checking if we can configure `docker-compose` version used.
Questions @j143 :
I do not know gitpod that much, but is there a way we could configure some "options" when starting such vm? for example it would be great if when starting the vm you could choose:
* backend
* python version
* integrations enabled
For integrations - maybe just some predefined sets of those would be enough: (`--integrations all` switch)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979298344
How about the comments with `--db-reset` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-982649747
Thank you @uranusjr for review. 😺
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j143 commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
j143 commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979333874
>I think maybe (not 100% sure if you think it's a good idea) you shoud add --db-reset in the command starting breeze in the gitpod configuration (./breeze --db-reset -y).
Is it ok, if I skip this note and add it as a new task in the #16480 . I need to spend little more time in this to understand this better.
But, If you suggesting adding `./breeze --db-reset -y`. I will do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19756: gitpodify Apache Airflow - online development workspace
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19756:
URL: https://github.com/apache/airflow/pull/19756#issuecomment-979329162
> please provide any comments on that. :)
I think maybe (not 100% sure if you think it's a good idea) you shoud add `--db-reset` in the command starting breeze in the gitpod configuration (`./breeze --db-reset -y`) . This way you could simply remove the instruction about "running airlfow db init once" - as the db will be initialized when you enter breeze. This has the drawback that every time you enter it, the db will be cleaned, but I think it happens anyway - I do not know for how long the remote VMs are cached and whether you always recreate it when you reconnect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org