You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/08 13:39:42 UTC
[GitHub] [airflow] potiuk opened a new pull request #22082: Add Codespaces support
potiuk opened a new pull request #22082:
URL: https://github.com/apache/airflow/pull/22082
Codespaces is (soon to be GA) feature that allows to start
development environment on a virtual machine directly from the
GitHub UI or from VSCode connected with GitHub account.
This PR adds configuration that allows to start codespaces
environment (based on Breeze's docker compose environment)
in the basic form that allows the users to run basic unit
tests.
It does not replace Breeze/CI but for most users who want
to just run a few tests, make sure that all dependencies
and plugins are installed so that the users can get the
VSCode autocomplete and execution of the tests possible.
This is a great way to begin your journey with Airflow - while
the Codespace machines are not very powerful and the integration
tests might require more resources, the "no-setup"
environment and no need to have powerful machine is great
for contributors who cannot afford powerful development
workstations.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1061794479
This is a very basic suppor for codespaces, but it works nicely.
For those who have access to codespaces (it's not yet GA for Apache and Public repos) it takes about 4 minutes to get you up and running:
You land inside "breeze" container with terminal and pytest working, postgres database and you can run unit tests from within VSCode (either in-browser or local).
It required to move the helm charts to "tests/charts folder" because of the way how VSCode does NOT handle well mutliple folders with pytest conftest.py in separate "test trees".
This is a small inconvenience (if at all) but I wanted you to take a look @jedcunningham @kaxil @ephraimbuddy.
Also as a side effect of it you can run just `pytest` in the root folder of airflow and all tests (including the helm chart tests) will get executed (and conftest.py will be properly applied to the subset of tests in each subfolder). This brings our test structure more in line with the "standard" pytest test tree approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062055640
> pretty cool +1
It is indeed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062096912
> > pretty cool +1
>
> BTW. @dstandish For me this is a culmination of 3.5 years of devenv work.
>
> All the Breeze, constraint, ci docker image etc. All of it was there to eventually enable this one seamlessly (I knew it was coming). This builds on top of all the automation that is behind maintaining our CI images and constraints updated so that whenever you start codespace on a new PR, you will get the latest and greatest environment to work on in < 10 minutes (we got down to < 3).
>
> This is beyond cool (but let's see how it is going to be useful - it is still not GA for all contributors unfortunately. It's only available as GA for Teams and GitHub Enterprise Cloud customers (and those users like me who applied for Beta access). When and how it will be available for personal repos/Open Source repos - we do not know (but I am meeting my friend in Berlin next week who works there and I hope I will know more :) .
yes, "pretty cool" does feel quite inadequate in hindsight :)
it's quite a mountain that you have climbed. i remember the days of the puckel image and when breeze was still being incubated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1061991972
I hope it will be better. Also @jedcunningham @kaxil @ephraimbuddy @ashb -> you might get happy with that change, as I also enabled parallel tests for the Helm Chart tests in this change.
It will immensely speed up the Helm chart Job I thinlk. Even on the super-fast machines we have in Self-Hosted runs it took 25 minutes to run. On my machine when I enabled parallel execution it took < 8 minutes so we might have a huge, huge, huge speedup. :crossed_fingers:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062079282
Right. I got it green now.
Please @jedcunningham @ephraimbuddy @kaxil take a look at the move of tests and whether the move of tests is "OK" for you,
The added benefit of this move is that integration with any other IDE for tests is now a BREEZE :). Tests will be automatically recognized and configured, without any special setup or special projects (i actually had to have previously multiple projects in my IDE to be able to run Helm tests - this is not needed any more.
Should be much easier to start their journey with Airflow now :)
BTW. I am not sure if you have codespaces or not... But if you do .... Simply press "." on this PR to try it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062083156
Ah no. Needs to be merged to `main` in order to apply for `apache/airflow` repo :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062071938
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062056334
Might make me switch to VSCode actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1061796461
I think it is totally worth it - especially for new contributors. My goal of < 10 minutes to get Airflow environment up-and-running is well exceeded with that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062089274
> pretty cool +1
BTW. @dstandish For me this is a culmination of 3.5 years of devenv work.
All the Breeze, constraint, ci docker image etc. All of it was there to eventually enable this one seamlessly (I knew it was coming). This builds on top of all the automation that is behind maintaining our CI images and constraints updated so that whenever you start codespace on a new PR, you will get the latest and greatest environment to work on in < 10 minutes (we got down to < 3).
This is beyond cool (but let's see how it is going to be useful - it is still not GA for all contributors unfortunately. It's only available as GA for Teams and GitHub Enterprise Cloud customers (and those users like me who applied for Beta access). When and how it will be available for personal repos/Open Source repos - we do not know (but I am meeting my friend in Berlin next week who works there and I hope I will know more :) .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062053159
Update: Setting up a working test-ready Codespace for airlfow takes ~ 3 minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062062530
> you a pycharm user? i've tried it out now and then... always run back to pycharm.... but will have to try this
Intellij Ultimate for many years. But recent event made me rethink where my money go, so even if this is my main productivity tool for years (except `vi` of course which is probably more so), VSCode is great alternative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1061997918
Ah no.... they were already enabled as auto. I am not sure why they take 3 times more in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1061933022
Need to work a bit on executing the tests on CI :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#discussion_r822121869
##########
File path: CONTRIBUTING.rst
##########
@@ -472,25 +481,27 @@ develop Apache Airflow:
The table below summarizes differences between the two environments:
Review comment:
```suggestion
The table below summarizes differences between the environments:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062861044
Merged!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #22082:
URL: https://github.com/apache/airflow/pull/22082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062057772
> Might make me switch to VSCode actually.
you a pycharm user? i've tried it out now and then... always run back to pycharm.... but will have to try this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#discussion_r822085697
##########
File path: CONTRIBUTING.rst
##########
@@ -197,15 +197,24 @@ From the `apache/airflow <https://github.com/apache/airflow>`_ repo,
Step 2: Configure Your Environment
----------------------------------
-You can use either a local virtual env or a Docker-based env. The differences
-between the two are explained `here <https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#development-environments>`__.
+You can use use several development environments for Airflow. If you prefer to have development environments
Review comment:
```suggestion
You can use several development environments for Airflow. If you prefer to have development environments
```
##########
File path: scripts/ci/libraries/_build_images.sh
##########
@@ -598,7 +598,7 @@ function build_images::build_prod_images() {
"--cache-from=${AIRFLOW_PROD_IMAGE}:cache"
)
else
- echo
+ echod
Review comment:
```suggestion
echo
```
Typo?
##########
File path: CONTRIBUTING.rst
##########
@@ -472,25 +481,27 @@ develop Apache Airflow:
The table below summarizes differences between the two environments:
-========================= ================================ =====================================
-**Property** **Local virtualenv** **Breeze environment**
-========================= ================================ =====================================
-Test coverage - (-) unit tests only - (+) integration and unit tests
-------------------------- -------------------------------- -------------------------------------
-Setup - (+) automated with breeze cmd - (+) automated with breeze cmd
-------------------------- -------------------------------- -------------------------------------
-Installation difficulty - (-) depends on the OS setup - (+) works whenever Docker works
-------------------------- -------------------------------- -------------------------------------
-Team synchronization - (-) difficult to achieve - (+) reproducible within team
-------------------------- -------------------------------- -------------------------------------
-Reproducing CI failures - (-) not possible in many cases - (+) fully reproducible
-------------------------- -------------------------------- -------------------------------------
-Ability to update - (-) requires manual updates - (+) automated update via breeze cmd
-------------------------- -------------------------------- -------------------------------------
-Disk space and CPU usage - (+) relatively lightweight - (-) uses GBs of disk and many CPUs
-------------------------- -------------------------------- -------------------------------------
-IDE integration - (+) straightforward - (-) via remote debugging only
-========================= ================================ =====================================
+========================= ================================ ===================================== ========================================
Review comment:
The line at 481 ` two environments:` can be changed to `three environments`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #22082:
URL: https://github.com/apache/airflow/pull/22082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1061796461
I think it is totally worth it - especially for new contributors. My goal of < 10 minutes to get Airflow dev environment up-and-running is well exceeded with that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #22082: Add Codespaces support
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #22082:
URL: https://github.com/apache/airflow/pull/22082#issuecomment-1062912281
Nice, well done. I haven't looked at the exact code changes but reading by the PR description, this looks great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org