You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/02 20:14:07 UTC
[GitHub] [airflow] potiuk opened a new pull request #19992: Use `python -m pip` to run pip in the new Breeze2
potiuk opened a new pull request #19992:
URL: https://github.com/apache/airflow/pull/19992
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987684134
Personally I think `pipx run` is good enough for the auto-upgrade use case. There is a run cache, but it gets purged pretty frequently (and you can always self-purge). But keeping both for a while is not a bad idea if you don’t feel comfortable with it yet. I think you eventually will though 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-986698026
> Did you move Breeze2.exe to another place to run it?
@edithturn - yep. I moved it to main folder of Airflow. This is where it shoudl be eventually if someone wants to run it - because "dist" is a temporary folder that gets overwritten.
@uranusjr - Actually I just wrote the ADR (I described all the decisions, reasonsing and assumptions in this PR).
I REALLY do not want to use any 3rd-party tools here which would be prerequisites for running Breeze (and they would not provide the same experience regarding the 1. `git checkout`. 2. Run.
However while preparing the ADR and looking closely at the alternatives I found out that `pipx` (which I understand is standard pypa project) also has editable mode option whic works ALMOST the same way as the automated bootstrapping of ours.
Ny proposal is to offer both alternatives to the users.
PIPX has two drawbacks:
* it will not automatically "upgrade when needed" which I think is an important property - also for the kinds of "gitpodify" and "codespace" environments where automated dependency management is really nice feature to have.
* when you have more than one airflow working copy, only one of those can be installed and used automatically at a time
For the next several months (at least - likely years) Breeze will be a living project, changing together with airflow code, and this
is quite a limitation - people willl be switching to different branches (2.3, 2.4 etc.) and there the Booststrapping script of ours has the benefit that it will always use "local" version of Breeze, not the one "globally installed".
So I think the best will be to leave our Breeze2 bootstrapping script for "no requirements needed", "local" experience, but also have `pipx install -e dev/breeze` as an alternative way of installing Breeze globally (especially if the user has only one checked out Airflow version, that would be almost equivalent to the Bootstrapping script of ours).
And seems that `pipx install -e dev\breeze` should work nicely and produce the right Windows experience too (@edithturn can you please doube check it)?
I think having both options (our Bootstrapping script and pipx option) is best of both worlds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987015592
Just a thought - maybe we could contribute the "auto-upgrade" feature to it and then we could indeed get rid of the bootstrapping script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769574843
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
(And from earielr looking at venv code, I believe it executes ensurepip under the hood).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-989825317
@uranusjr - shall we ? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769580862
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
I think it's as good as it gets - there is no need to handle that in any other way, I think, if the error is that detailed and gives you that explanatory message, I think we should be good with that, but if you think there are other cases where the message will not be good, we can actually check it and write our own message - do you think there are other cases that the message might be misleading? I am all for making it as clear to our users as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-985087637
> Ah - so maybe on windows we have `Scripts` instead of bin ? @uranusjr ?
There is Python310/Scripts, where pip is, and I added that path into Variable environments to use pip and python. (not sure if we are speaking about the same directory)
Also I used python for 64 bits and pyinstaller of 64 bits. ( they have to be similar to build something without that message of virus)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-986755223
`@edithturn - yep. I moved it to main folder of Airflow. This is where it shoudl be eventually if someone wants to run it - because "dist" is a temporary folder that gets overwritten.
Yes, I also tried in the airflow folder, giving me the result for paths that don't find`
Let me read and try this: "pipx install -e dev\breeze"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987014257
BTW @uranusjr - `pipx` is REALLY cool. as discussed with Edith on slack :). Almost passes my Bar to be used instead of the Boostrapping script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-986455464
> > I think it makes little sense to add the .exe to repo (it's quite big) and everyone should be able to generate one themselves if they need it - with simple one-liner (well maybe we can add a .bat script for that that will install pyinstaller, prepare the binary and move it to Airflow repo?)
>
> Yes, having a .bat and instructions will be really great once It passed some tests more.
There’s a reason pip does not use a Bat, and the shortest version of that is Bat scripts don’t work as well as you may imagine outside of the simplest of scenarios. Stick to a real executable and you will not regret it.
If the goal is to eliminate infrastructure, @vsajip maintains a set of “simple launchers” we can use to create the shims: https://bitbucket.org/vinay.sajip/simple_launcher/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-984967036
cc: @edithturn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-986393525
> I think it makes little sense to add the .exe to repo (it's quite big) and everyone should be able to generate one themselves if they need it - with simple one-liner (well maybe we can add a .bat script for that that will install pyinstaller, prepare the binary and move it to Airflow repo?)
Yes, having a .bat and instructions will be really great once It passed some tests more.
> I also noticed that whe you install Git on Windows, you can also choose to install "Git Bash" (based on MinGW). And I think this provides another nice option for Windows users - Git is another "implicit" prerequisite and we might simply ask the users to choose this option when installing Git. Then the ./Breeze2 command works exactly the same as on Linux/MacOS.
Yes, on Git Bash we should just run ./Breeze2.exe. I was using it, we can run some Linux commands like: "ls", very helpful if we don't remember cmd commands. We should definitely add it as an "implicit" requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987014576
And it seamlessly works on Windows too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-986390697
I got it @potiuk I tested it again, i didn't give me the same result as you:
- git clone https://github.com/potiuk/airflow.git
- git checkout use-python-m-to-run-pip-new-breeze
- python Breeze2:
- This installed all the dependencies into ...airflow/dev/breeze
- Geet requirements to build wheel, pyparsing, py,,, etc
- Installed Breeze's virtual env in airflow\.build\breeze2\venv
- pyinstaller Breeze2
- cd dist, ./Breeze2/Breeze2.exe and I got this message:
`[196] Failed to execute script 'Breeze2' due to unhandled exception!
Traceback (most recent call last):
File "Breeze2", line 45, in <module>
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['C:\\Users\\Marcia\\Documents\\EdithPuclla\\repo\\airflow\\dist\\Breeze2\\Breeze2.exe', '-m', 'venv', 'C:\\Users\\Marcia\\Documents\\EdithPuclla\\repo\\airflow\\dist\\Breeze2\\.build\\breeze2\\venv']' returned non-zero exit status 1.`
Not sure If I am doing something wrong on my environment, I assumed that I should test your branch from your personal Github, but I also test main on airflow again.
Did you move Breeze2.exe to another place to run it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769347765
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
Also the `shell=True` part is unnecessary and only adds overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769576095
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
This is also what the documentation says:
> Unless the --without-pip option is given, ensurepip will be invoked to bootstrap pip into the virtual environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-985010852
I checked that path: .build\breeze2\venv\bin on windows installation, "bin doesn't exist", in my test the installations are not generating that directory. I just got these files/directories instead on the venv directory:
- Include
- lib
- Scripts
- pyenv
In Linux might be a different behavior to generate that venv directory. I am not sure. The error I was getting was this: "...dist\\Breeze2\\.build\\breze2\\venv returned non-zero exit status [4712] Failed to execute script 'Breeze2' due to unhandled exception"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-985081675
Ah - so maybe on windows we have `Scripts` instead of bin ? @uranusjr ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-985489503
> The sysconfig module provides the programmatic interface to the directory structure - I'd suggest using "sysconfig.get_paths()" to find the locations where files may be written to.
I am a little confused what should be the "locations"?
And for this:
> "env_dir: for creating a virtual environment by specifying the target directory"
What should be our target directory? it should be .build?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r762527267
##########
File path: Breeze2
##########
@@ -41,11 +43,21 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
)
+ if os.name == 'nt':
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}.exe", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
+ else:
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
Review comment:
Cool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-991743821
@uranusjr ? Is that one cool to go ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769573884
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
BTW. I was under the impression that as of 3.3 venv is part of the standard library? At least that's what https://docs.python.org/3.7/library/venv.html suggests. Am I wrong about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769347325
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
Not convinced we should do this, but I’m not going to spend time arguing about it. I’m pretty confident you’ll regret this when people come complaining this failed on them (because `python3` points to a system Python for them and they didn’t install `python3-venv` or ensurepip).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-995821982
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987915375
> Personally I think `pipx run` is good enough for the auto-upgrade use case. There is a run cache, but it gets purged pretty frequently (and you can always self-purge). But keeping both for a while is not a bad idea if you don’t feel comfortable with it yet. I think you eventually will though slightly_smiling_face
It's very likely actually - when we get to the point that we will stop adding new dependencies and new command line options to Breeze2. I just REALLY want to avoid question in #breeze channel "I switched to 2.6 branch (latest stable) and I want to test something and my Breeze2 stopped working".
Once we get to the point when single Breeze version will work fine for both `main` and latest stable branch" I'd definitely switch to `pipx` and we would even likely be able to remove bootstraping script and then even likely we might push `apache-airflow-breeze` as package as you proposed. But not sooner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-985099912
Hmm. And we should really create the venv with the API. that will be even more portable. Rather than running "python -m venv, we could simply use: https://docs.python.org/3/library/venv.html#api
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r762529419
##########
File path: Breeze2
##########
@@ -41,11 +43,21 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
)
+ if os.name == 'nt':
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}.exe", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
+ else:
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
Review comment:
OK. Got Windows installed in VirtualBox now so I will do seme more tests with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-986393525
> I think it makes little sense to add the .exe to repo (it's quite big) and everyone should be able to generate one themselves if they need it - with simple one-liner (well maybe we can add a .bat script for that that will install pyinstaller, prepare the binary and move it to Airflow repo?)
Yes, having a .bat and instructions will be really great once It passed some tests more.
> I also noticed that whe you install Git on Windows, you can also choose to install "Git Bash" (based on MinGW). And I think this provides another nice option for Windows users - Git is another "implicit" prerequisite and we might simply ask the users to choose this option when installing Git. Then the ./Breeze2 command works exactly the same as on Linux/MacOS.
Yes, on Git Bash we should just run ./Breeze2.exe. I was using it, we can run some Linux commands like: "ls", very helpful if we don't remember cmd commands. We should definitely add it as an "implicit" requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-995825466
@uranusjr - discussed it on todays "outreachy" meeting - I am merging that one with @eladkal's approval - happy to discuss it later and improve and possibly finallly go the `pipx` way as we get to stability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987109970
Hey @uranusjr after some pondering I updated the ADR to make it clear that both - bootstrapping script and `pipx` are good ways to install Breeze (with the small caveats of pipx)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r762525565
##########
File path: Breeze2
##########
@@ -41,11 +43,21 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
)
+ if os.name == 'nt':
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}.exe", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
+ else:
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
Review comment:
Windows appends the executable extension automatically, so you don’t need to append `.exe` on your own. The behaviour is not exactly the same—it uses `PATHEXT`, which contains more than just `.exe`, so theoratically this may pick up a `python.bat`—but only with negligible difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-986255818
I think I found the combination that works (and @edithturn - it was not easy :). But it should nicely work now - both when the "Bootstrap" is run via `python Breeze2` and via "Bundled" Breeze2.exe. You can now try to build the .exe file and see if that works in your setup as well, and once it does, we might simply merge this one and maybe you could test it and add some instructions on how to build such a binary (and avold all the "Defender") problems as well.
I think it makes little sense to add the .exe to repo (it's quite big) and everyone should be able to generate one themselves if they need it - with simple one-liner (well maybe we can add a .bat script for that that will install pyinstaller, prepare the binary and move it to Airflow repo?)
I also noticed that whe you install Git on Windows, you can also choose to install "Git Bash" (based on MinGW). And I think this provides another nice option for Windows users - Git is another "implicit" prerequisite and we might simply ask the users to choose this option when installing Git. Then the `./Breeze2` command works exactly the same as on Linux/MacOS.
I think this all combined is "good enough" for the Windows users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769571893
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
Is there a better option @uranusjr ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987986808
@uranusjr - I changed the name of the PR to reflect the dual nature of running PIP. I also added autoamated detection of where the airflow sources are:
1) First we look in the working directory (upwards)
2) If we are outside of working directory of Airflow, we look into where it was installed from (upwards)
This does not support the case that we want to run (potentially) different Breeze depending on where we run it (which our bootstrap script does) but it handles a different case (which `pipx` is actually better at) that Breeze2 might be run from any directory, even outside of Airflow code and it will work `as if` it was run from withing the repo.
@edithturn - we are close to merging it, I think :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987986808
@uranusjr - I changed the name of the PR to reflect the dual nature of running PIP. I also added autoamated detection of where the airflow sources are:
1) First we look in the working directory
2) If we are outside of working directory of Airflow, we look into where it was installed from
This does not support the case that we want to run (potentially) different Breeze depending on where we run it (which our bootstrap script does) but it handles a different case (which `pipx` is actually better at) that Breeze2 might be run from any directory, even outside of Airflow code and it will work `as if` it was run from withing the repo.
@edithturn - we are close to merging it, I think :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987986808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-991743821
@uranusjr ? Is that one cool to go ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-993859118
@uranusjr :) We neeed it for @Bowrna :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19992:
URL: https://github.com/apache/airflow/pull/19992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-986100524
Try it now @edithturn on Windows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r762527754
##########
File path: Breeze2
##########
@@ -41,11 +43,21 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
)
+ if os.name == 'nt':
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}.exe", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
+ else:
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
Review comment:
Actually I want to add back `python -m venv` as fallback. I noticed that I had problems with making the API works on MacOS when I had default MacOS Python set as "global" Python rather than the one coming from Pyenv. Configuring Pyevn to work properly on MacOS is kinda not obvious in this case.
It got fixed when I set `pyenv global`
The problem I had was that "ensurepip" was missing.
Is it a known issue @uranusjr ? I might want to reproduce it and possibly report it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xurror commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
xurror commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r767714020
##########
File path: dev/breeze/doc/adr/0003-bootstraping-virtual-environment.md
##########
@@ -0,0 +1,200 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [3. Bootstraping the virtual environment](#3-bootstraping-the-virtual-environment)
+ - [Status](#status)
+ - [Context](#context)
+ - [Decision](#decision)
+ - [Alternatives considered](#alternatives-considered)
+ - [Consequences](#consequences)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# 3. Bootstraping the virtual environment
+
+Date: 2021-12-06
+
+## Status
+
+Draft
+
+## Context
+
+Since Breeze is written in Python, it needs to be run in its own virtual environment.
+This virtual environment is different from Airflow virtualenv as it contains only a
+small set of tools (for example rich) that are not present in the standard Python
+library. We want to keep the virtualenv separated, because setting up Airflow
+virtualenv is hard (especially if you consider cross-platform use). The virtualenv
+is needed mainly to run the script that will actually manage airflow installation
+and dependencies, in the form of Docker images which are part of Breeze.
+
+This virtualenv needs to be easy to setup and it should support the "live" nature
+of Breeze. The idea is that the user of Breeze does not have to do any action
+to update to the latest version of the virtualenv, when new dependencies are
+added, also when new Breeze functionalities are added, they should be automatically
+available for the user after the repository is updated to latest version.
+
+User should not have to think about installing and upgrading Breeze separately from
+switching to different Airflow tag or branch - moreover, the Breeze environment
+should automatically adapt to the version and Branch the user checked out. By its
+nature Airflow Breeze (at least for quite a while) will be evolving together with
+Airflow and it will live in the same repository and new features and behaviours
+will be added continuously.
+
+The workflow that needs to be supported should tap into the regular workflow
+of the user who is developing Airflow.
+
+* git checkout branch
+
+./Breeze should use the version of Breeze that is available in this version
+
+* git rebase --onto apache/main
+
+./Breeze should be automatically updated to the latest version available
+in main (including dependencies)
+
+Also if someone develops Breeze itself, the experience should be seamlessly
+integrated - modification of Breeze code locally should be automatically
+reflected in the Breeze environment of the user who is modifying Breeze.
+
+The user should not have to re-install/update Breeze to automatically use
+the modifying Breeze source code when running Breeze commands and testing
+then with Airflow.
+
+Breeze is also used as part of CI - common Python functions and libraries
+are used across both Breeze development environment and Continuous
+Integration we run. It's been established practice of the CI is that the logic
+of the CI is stored in the same repository as the source code of the
+application it tests and part of the Breeze functions are shared with CI.
+
+In the future when Breeze2 stabilizes and it's update cadence will be
+much slower (which is likele as it happened with the Breeze predecessor)
+there could be an option that Breeze is installed as separate package and
+same released Breeze version could be ued to manage multiple Airflow
+versions, for that we might want to release Breeze as a separate package
+in PyPI. However since there is the CI integration, the source code
+version of Breeze will remain as part of the Airflow's source code.
+
+
+## Decision
+
+The decision is to implement Breeze in a subfolder (`dev/breeze2/`) of
+Apache Airflow as a Python project following the standard setuptools
+enabled project. The project contains setup.py and dependencies described
+in setup.cfg and contains both source code and tests for Breeze code.
+
+The sub-project could be used in the future to produce a PyPI package
+(we reserved such package in PyPI), however its main purpose is
+to install Breeze in a separate virtualenv bootstrapped
+automatically in editable mode.
+
+There are two ways you will be able to install `Breeze2` - locally in
+repository using ./Breeze2 bootstrapping script and using `pipx`.
+
+The bootstrapping Python script (`Breeze2` in the main repository
+of Airflow performs the following tasks:
+
+* when run for the first time it creates `.build/breeze2/venv` virtual
+ environment (Python3.6+ based) - with locally installed `dev`
+ project in editable mode (`pip install -e .`) - this makes sure
+ that the users of Breeze will use the latest version of Breeze
+ available in their version of the repository
+* when run subsequently, it will check if setup files changed for
+ Breeze (dependencies changed) and if they did it will automatically
+ reinstall the environment, adding missing dependencies
+* after managing the venv, the Breeze2 script will simply execute
+ the actual Breeze2 script in the `.build/venv` passing the
+ parameters to the script. For the user, the effect will be same
+ as activating the virtualenv and executing the ./Breeze2 from
+ there (but it will happen automatically and invisibly for the
+ user
+* In Windows environment where you have no easy/popular mechanism
+ of running scripts with shebang (#!) equivalent in Posix
+ environments, Users will be able to locally build (using
+ `pyinstaller` a `Breeze2.exe` frozen Python script that will
+ essentially do the same, they could also use `python Breeze2`
+ command or switch to Git Bash to utilize the shebang feature
+ (Git Bash comes together with Git when installed on Windows)
+* The second option is to use `pipx` to install Breeze2.
+ `Pipx` is almost equivalent to what the Bootstrapping does
+ and many users might actually choose to install Breeze this
+ way - and we will add is as an option to install Breeze
Review comment:
Another typo: we will add `is` as an option... instead of we will add `it` as an option...
##########
File path: dev/breeze/doc/adr/0003-bootstraping-virtual-environment.md
##########
@@ -0,0 +1,200 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [3. Bootstraping the virtual environment](#3-bootstraping-the-virtual-environment)
+ - [Status](#status)
+ - [Context](#context)
+ - [Decision](#decision)
+ - [Alternatives considered](#alternatives-considered)
+ - [Consequences](#consequences)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# 3. Bootstraping the virtual environment
+
+Date: 2021-12-06
+
+## Status
+
+Draft
+
+## Context
+
+Since Breeze is written in Python, it needs to be run in its own virtual environment.
+This virtual environment is different from Airflow virtualenv as it contains only a
+small set of tools (for example rich) that are not present in the standard Python
+library. We want to keep the virtualenv separated, because setting up Airflow
+virtualenv is hard (especially if you consider cross-platform use). The virtualenv
+is needed mainly to run the script that will actually manage airflow installation
+and dependencies, in the form of Docker images which are part of Breeze.
+
+This virtualenv needs to be easy to setup and it should support the "live" nature
+of Breeze. The idea is that the user of Breeze does not have to do any action
+to update to the latest version of the virtualenv, when new dependencies are
+added, also when new Breeze functionalities are added, they should be automatically
+available for the user after the repository is updated to latest version.
+
+User should not have to think about installing and upgrading Breeze separately from
+switching to different Airflow tag or branch - moreover, the Breeze environment
+should automatically adapt to the version and Branch the user checked out. By its
+nature Airflow Breeze (at least for quite a while) will be evolving together with
+Airflow and it will live in the same repository and new features and behaviours
+will be added continuously.
+
+The workflow that needs to be supported should tap into the regular workflow
+of the user who is developing Airflow.
+
+* git checkout branch
+
+./Breeze should use the version of Breeze that is available in this version
+
+* git rebase --onto apache/main
+
+./Breeze should be automatically updated to the latest version available
+in main (including dependencies)
+
+Also if someone develops Breeze itself, the experience should be seamlessly
+integrated - modification of Breeze code locally should be automatically
+reflected in the Breeze environment of the user who is modifying Breeze.
+
+The user should not have to re-install/update Breeze to automatically use
+the modifying Breeze source code when running Breeze commands and testing
+then with Airflow.
+
+Breeze is also used as part of CI - common Python functions and libraries
+are used across both Breeze development environment and Continuous
+Integration we run. It's been established practice of the CI is that the logic
+of the CI is stored in the same repository as the source code of the
+application it tests and part of the Breeze functions are shared with CI.
+
+In the future when Breeze2 stabilizes and it's update cadence will be
+much slower (which is likele as it happened with the Breeze predecessor)
+there could be an option that Breeze is installed as separate package and
+same released Breeze version could be ued to manage multiple Airflow
+versions, for that we might want to release Breeze as a separate package
+in PyPI. However since there is the CI integration, the source code
+version of Breeze will remain as part of the Airflow's source code.
+
+
+## Decision
+
+The decision is to implement Breeze in a subfolder (`dev/breeze2/`) of
+Apache Airflow as a Python project following the standard setuptools
+enabled project. The project contains setup.py and dependencies described
+in setup.cfg and contains both source code and tests for Breeze code.
+
+The sub-project could be used in the future to produce a PyPI package
+(we reserved such package in PyPI), however its main purpose is
+to install Breeze in a separate virtualenv bootstrapped
+automatically in editable mode.
+
+There are two ways you will be able to install `Breeze2` - locally in
+repository using ./Breeze2 bootstrapping script and using `pipx`.
+
+The bootstrapping Python script (`Breeze2` in the main repository
+of Airflow performs the following tasks:
+
+* when run for the first time it creates `.build/breeze2/venv` virtual
+ environment (Python3.6+ based) - with locally installed `dev`
+ project in editable mode (`pip install -e .`) - this makes sure
+ that the users of Breeze will use the latest version of Breeze
+ available in their version of the repository
+* when run subsequently, it will check if setup files changed for
+ Breeze (dependencies changed) and if they did it will automatically
+ reinstall the environment, adding missing dependencies
+* after managing the venv, the Breeze2 script will simply execute
+ the actual Breeze2 script in the `.build/venv` passing the
+ parameters to the script. For the user, the effect will be same
+ as activating the virtualenv and executing the ./Breeze2 from
+ there (but it will happen automatically and invisibly for the
+ user
+* In Windows environment where you have no easy/popular mechanism
+ of running scripts with shebang (#!) equivalent in Posix
+ environments, Users will be able to locally build (using
+ `pyinstaller` a `Breeze2.exe` frozen Python script that will
+ essentially do the same, they could also use `python Breeze2`
+ command or switch to Git Bash to utilize the shebang feature
+ (Git Bash comes together with Git when installed on Windows)
+* The second option is to use `pipx` to install Breeze2.
+ `Pipx` is almost equivalent to what the Bootstrapping does
+ and many users might actually choose to install Breeze this
+ way - and we will add is as an option to install Breeze
+ with pipx `pipx install -e <BREEZE FOLDER>` provides the right
+ installation instruction. The installation can be updated
+ by `pipx install --force -e <BREEZE FOLDER>`.
+ The benefit of using `pipx` is that Breeze becomes
+ available on the path when you install it this way, also
+ it provides out-of-the box Windows support. The drawback is
+ that when new dependencies are added, they will not be
+ automatically installed and that you need to manually force
+ re-installation if new dependencies are used - which is not
+ as seamlessly integrate in the regular development
+ environment, and it might create some confusions for the
+ users who would have to learn `pipx` and it's commands.
+ Another drawback of `pipx` is that installs one global
+ version of Breeze2 for all projects, where it is quite
+ possible that someone has two different versions of
+ Airflow repository checked out and the bootstraping
+ script provides this capability.
+
+## Alternatives considered
+
+The alternatives considered were:
+
+* nox - nox is a tool to manage virtualenv for testing, while
+ it has some built in virtualenv capabilities, it is an
+ additional tool that needs to be installed and it lacks
+ the automation of checking and recreation of the virtualenv
+ when needed (you need to manually run nox to update environment)
+ Alsoi it is targeted for building multiple virtualenv
+ for tests - it has nice pytest integration for example, but it
+ lacks support for managing editable installs for a long time.
+
+* pyenv - this is the de-facto standard for maintenance of
+ virtualenvs. it has the capability of creation and switching
+ between virtualenvs easily. Together with some of its plugins
+ (pyenv-virtualenv and auto-activation) it could serve the
+ purpose quite well. However the problem is that if you
+ also use peynv to manage your `airflow` virtualenv this might
Review comment:
Little typo on this line, you wrote `peynv` instead of `pyenv`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769581162
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
BTW. shell is removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r769578989
##########
File path: Breeze2
##########
@@ -41,15 +49,35 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
- )
+ try:
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
+ )
+ except Exception as e:
+ # in some cases (mis-configured python) the venv creation might not work via API
+ # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
+ # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
+ # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
+ # interpreted executable available, because Python interpreter is executed through a library.
+ # and sys.executable points to the Bundled exe file.
+ BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
+ subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True, shell=True)
Review comment:
Ah ok - I think you refer to this error : https://stackoverflow.com/questions/39539110/pyvenv-not-working-because-ensurepip-is-not-available - well, then I think the error message is rather clear:
```
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt-get install python3-venv
You may need to use sudo with that command. After installing the python3-venv
package, recreate your virtual environment.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r762536434
##########
File path: Breeze2
##########
@@ -41,11 +43,21 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
)
+ if os.name == 'nt':
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}.exe", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
+ else:
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
Review comment:
Not with venv (it is very barebone), but if you use virtualenv there is a thing called [seeders](https://virtualenv.pypa.io/en/latest/cli_interface.html#section-seeder).
Installing things on environment creation is out of pip’s control because it needs hooks in the environment builder, and venv does not provide that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-985087637
> Ah - so maybe on windows we have `Scripts` instead of bin ? @uranusjr ?
There is Python/Scripts, where pip is, and I added that path into Variable environments to use pip and python. (not sure if we are speaking about the same directory)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r762586811
##########
File path: Breeze2
##########
@@ -41,11 +43,21 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
)
+ if os.name == 'nt':
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}.exe", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
+ else:
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
Review comment:
Some more info: the main problem was "trying" to create venv from within the "frozen" python script (in bundled .exe file). I debugged it quite a bit and I've learned quite a lot about those things.
The "frozen" python binary does not run using executable interpreter. This was a bit surprising - it actually uses a python interpereter library, which you cannot "execute" modules with. And `sys.executable` is surprisingly the actual .exe file that was run :scream: (so in our case ./Breeze2.exe). @edithturn - that's why you got those repeating "creating environement| messages - the python code simply continued to re-run the `Breeze2.exe` on failure of trying to create an evnironment using the API.
And creating using the API failed for a different reason. First of all ensurepip is run "implicitly" by "venv create" and pip installer when prepares a bundle, only uses modules thata are actually used by the python script (and ensurepip was not detected).
But even if I used `---hidden-import` flag of the `pyinstaller` and added it (Pyinstaller is actually quite configurable), the `ensurepip` when installing PIP in the environment, also uses the python interpreter executable :scream: and of course gets into another endless loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987915375
> Personally I think `pipx run` is good enough for the auto-upgrade use case. There is a run cache, but it gets purged pretty frequently (and you can always self-purge). But keeping both for a while is not a bad idea if you don’t feel comfortable with it yet. I think you eventually will though slightly_smiling_face
It's very likely actually - when we get to the point that we will stop adding new dependencies and new command line options to Breeze2. I just REALLY want to avoid question in #breeze channel "I switched to 2.6 branch (latest stable) and I want to test something and my Breeze2 stopped working".
Once we get to the point when single Breeze version will work fine for both `main` and latest stable branch" I'd definitely switch to `pipx` and we would even likely be able to remove it and then even likely we might push `apache-airflow-breeze` as package as you proposed. But not sooner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987202065
Shall we @uranusjr ? I am happy later to revert to `pipx` once it will be able to support the two use-cases - auto-upgrade and having different app versions installed in different repos. It's kinda possible now, but in some cumbersome ways.
a) you can set PIPX_HOME to the `./.build` folder inside the repo (but then there is a question of "path" precedence
b) you can run `pip install --upgrade apache-airlfow-breezet` to upgrade an installation I think (not sure if the `-e` is preserved (but I think it should), or by runinng `pipx install --force -e dev/breeze`. I think it should be just nicer/more seamlessly integrated into running Breeze
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#discussion_r762528468
##########
File path: Breeze2
##########
@@ -41,11 +43,21 @@ def save_config():
if needs_installation():
print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run([sys.executable, "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PIP}", "install", "--upgrade", "-e", "."], cwd=BREEZE_SOURCE_PATH, check=True
+ EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
+ str(BUILD_BREEZE_VENV_DIR)
)
+ if os.name == 'nt':
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}.exe", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
+ else:
+ subprocess.run(
+ [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
+ cwd=BREEZE_SOURCE_PATH,
+ check=True,
+ )
Review comment:
BTW. IS there an API (PIP PI I guess) I could also use to install the needed packages into the new venv? Kinda strange to use subprocess for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-985096482
> Ah - so maybe on windows we have Scripts instead of bin ? @uranusjr ?
Oh yeah:
https://bugs.python.org/issue42863
> Also I used python for 64 bits and pyinstaller of 64 bits. ( they have to be similar to build something without that message of virus)
Yep. I think we should stick to 64 bits as base platform. 32 is kinda really really old.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-985097296
> There is Python310/Scripts, where pip is, and I added that path into Variable environments to use pip and python. (not sure if we are speaking about the same directory)
From the issue:
> The sysconfig module provides the programmatic interface to the directory structure - I'd suggest using "sysconfig.get_paths()" to find the locations where files may be written to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-987915375
> Personally I think `pipx run` is good enough for the auto-upgrade use case. There is a run cache, but it gets purged pretty frequently (and you can always self-purge). But keeping both for a while is not a bad idea if you don’t feel comfortable with it yet. I think you eventually will though slightly_smiling_face
It's very likely actually - when we get to the point that we will stop adding new dependencies and new command line options to Breeze2. I just REALLY want to avoid question in #breeze channel "I switched to 2.6 branch (latest stable) and I want to test something and my Breeze2 stopped working".
Once we get to the point when single Breeze version will work fine for both `main` and latest stable branch" I'd definitely switch to `pipx` and we would even likely be able to remove bootstraping script and then even likely we might push `apache-airflow-breeze` to PyPI as package as you proposed. But not sooner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-992441122
> I mostly noticed typos while reading the doc but I want to try this locally on my windows and see if all works well or maybe also suggest improvements.
Fixed it. @uranusjr - I have also added paragraph when I stress that bootstrapping script is temporary and that we are going to consider removing it in favour of `pipx` when the dependencies will stabilize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Enable local Breeze script and pipx to be used for breeze bootstrap
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19992:
URL: https://github.com/apache/airflow/pull/19992#issuecomment-989274792
Looks like goood to merge !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org