You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/13 10:04:41 UTC
[GitHub] [airflow] joppevos opened a new pull request, #22971: add script to initialise virtualenv
joppevos opened a new pull request, #22971:
URL: https://github.com/apache/airflow/pull/22971
Script to initialise a local virtual environment to develop.
Issue: #21105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850281284
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -59,33 +65,34 @@ def main():
"""
Setup local virtual environment.
"""
- airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path(__file__).parents[4] / "airflow")
Review Comment:
I think that was ok before.
It's the "airflow sources" should be (just checked):
Path(__file__).absolute().parents[2]
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -59,33 +65,34 @@ def main():
"""
Setup local virtual environment.
"""
- airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path(__file__).parents[4] / "airflow")
Review Comment:
I think that was ok before.
It's the "airflow sources" should be (just checked):
```
Path(__file__).absolute().parents[2]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850274934
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -103,23 +110,24 @@ def main():
"sudo apt install build-essential python3-dev libsqlite3-dev openssl"
"sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql"
)
- return
+ sys.exit(4)
print("\nResetting AIRFLOW sqlite database...")
- os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False"
- os.environ["AIRFLOW__CORE__UNIT_TEST_MODE"] = "False"
- os.environ["AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED"] = "False"
- os.environ["AIRFLOW__CORE__DAGS_FOLDER"] = f"{airflow_sources}/empty"
- os.environ["AIRFLOW__CORE__PLUGINS_FOLDER"] = f"{airflow_sources}/empty"
- subprocess.run(["airflow", "db", "reset", "--yes"])
+ env = os.environ.copy()
+ env["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False"
+ env["AIRFLOW__CORE__UNIT_TEST_MODE"] = "False"
+ env["AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED"] = "False"
+ env["AIRFLOW__CORE__DAGS_FOLDER"] = f"{airflow_sources}/empty"
+ env["AIRFLOW__CORE__PLUGINS_FOLDER"] = f"{airflow_sources}/empty"
+ subprocess.run(["airflow", "db", "reset", "--yes"], env=env)
print("\nResetting AIRFLOW sqlite unit test database...")
- os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "True"
- os.environ["AIRFLOW__CORE__UNIT_TEST_MODE"] = "False"
- os.environ["AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED"] = "False"
- os.environ["AIRFLOW__CORE__DAGS_FOLDER"] = f"{airflow_sources}/empty"
- os.environ["AIRFLOW__CORE__PLUGINS_FOLDER"] = f"{airflow_sources}/empty"
- subprocess.run(["airflow", "db", "reset", "--yes"])
+ env["AIRFLOW__CORE__LOAD_EXAMPLES"] = "True"
Review Comment:
```suggestion
env = os.environ.copy()
env["AIRFLOW__CORE__LOAD_EXAMPLES"] = "True"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850293235
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
We should add shebang here (see other scripts):
```
#!/usr/bin/env python3
```
And make the script executable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22971:
URL: https://github.com/apache/airflow/pull/22971#issuecomment-1105160326
Finally got some time to take a look and check @joppevos. I have added a small fixup there with a few small improvements:
* There was a slight error in the docuemenation/script. In fact, in order to install airlfow from "main" with extras you need to use `constraints-source-providers-main" not `constraints-main`. This accounts for very specific case when the providers is already changed in main but not released yet. I tracked and updated all the places in the docs mentioning it - and fixed the script.
* I changed the default to "devel" - very rarely you need devel_all in local virtualenv and it is prone to errors and configuraiton in your host. So I switch default to `devel` and added information about other options. I think this is much better this way:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850284966
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -21,11 +21,17 @@
import sys
from pathlib import Path
+if __name__ not in ("__main__", "__mp_main__"):
+ raise SystemExit(
+ "This file is intended to be executed as an executable program. You cannot use it as a module."
+ f"To run this script, run the ./{__file__} command"
+ )
+
-def clean_up_airflow_sources(airflow_sources: Path):
- if airflow_sources.exists():
- print(f"Removing {airflow_sources}")
- shutil.rmtree(airflow_sources, ignore_errors=True)
+def clean_up_airflow_home(airflow_home: Path):
+ if airflow_home.exists():
+ print(f"Removing {airflow_home}")
+ shutil.rmtree(airflow_home, ignore_errors=True)
def check_if_in_virtualenv() -> bool:
Review Comment:
Few lines below there is the "devel-all" extra but it is actually wrong - it should be "devel_all" instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r849323695
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ return
+
+ print(f"\nWiping and recreating {airflow_home_dir}")
+
+ if not airflow_home_dir == airflow_sources:
+ print("AIRFLOW_HOME and Source code are in the same path")
+ print(
+ f"When running this script it will delete all files in path {airflow_home_dir} "
+ "to clear dynamic files like config/logs/db"
+ )
+ print("Please move the airflow source code elsewhere to avoid deletion")
+ return
+
+ clean_up_airflow_sources(airflow_home_dir)
+
+ print("\nInstalling requirements...")
+ return_code = pip_install_requirements()
+
+ if return_code != 0:
+ print("Error installing the requirements")
+ print("Try running the command below and rerun virtualenv installation\n")
+ os_type = sys.platform
+ if os_type == "darwin":
+ print("brew install sqlite mysql postgresql openssl")
+ print("export LDFLAGS=\"-L/usr/local/opt/openssl/lib\"")
+ print("export CPPFLAGS=\"-I/usr/local/opt/openssl/include\"")
+ else:
+ print(
+ "sudo apt install build-essential python3-dev libsqlite3-dev openssl"
+ "sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql"
+ )
+ return
+
+ print("\nResetting AIRFLOW sqlite database...")
+ os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False"
+ os.environ["AIRFLOW__CORE__UNIT_TEST_MODE"] = "False"
+ os.environ["AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED"] = "False"
+ os.environ["AIRFLOW__CORE__DAGS_FOLDER"] = f"{airflow_sources}/empty"
+ os.environ["AIRFLOW__CORE__PLUGINS_FOLDER"] = f"{airflow_sources}/empty"
+ subprocess.run(["airflow", "db", "reset", "--yes"])
Review Comment:
This is fine. We are using similar approach elsewhere and it's "good enough" here. In other places we also add a util that alllows to "dry-run" the command and "verbose" print it when executed, but this script should be as standalone as possible and it's rather simple command to run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joppevos commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
joppevos commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850128061
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
Review Comment:
good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850275338
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -103,23 +110,24 @@ def main():
"sudo apt install build-essential python3-dev libsqlite3-dev openssl"
"sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql"
)
- return
+ sys.exit(4)
print("\nResetting AIRFLOW sqlite database...")
- os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False"
- os.environ["AIRFLOW__CORE__UNIT_TEST_MODE"] = "False"
- os.environ["AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED"] = "False"
- os.environ["AIRFLOW__CORE__DAGS_FOLDER"] = f"{airflow_sources}/empty"
- os.environ["AIRFLOW__CORE__PLUGINS_FOLDER"] = f"{airflow_sources}/empty"
- subprocess.run(["airflow", "db", "reset", "--yes"])
+ env = os.environ.copy()
+ env["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False"
+ env["AIRFLOW__CORE__UNIT_TEST_MODE"] = "False"
+ env["AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED"] = "False"
+ env["AIRFLOW__CORE__DAGS_FOLDER"] = f"{airflow_sources}/empty"
+ env["AIRFLOW__CORE__PLUGINS_FOLDER"] = f"{airflow_sources}/empty"
+ subprocess.run(["airflow", "db", "reset", "--yes"], env=env)
print("\nResetting AIRFLOW sqlite unit test database...")
- os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "True"
- os.environ["AIRFLOW__CORE__UNIT_TEST_MODE"] = "False"
- os.environ["AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED"] = "False"
- os.environ["AIRFLOW__CORE__DAGS_FOLDER"] = f"{airflow_sources}/empty"
- os.environ["AIRFLOW__CORE__PLUGINS_FOLDER"] = f"{airflow_sources}/empty"
- subprocess.run(["airflow", "db", "reset", "--yes"])
+ env["AIRFLOW__CORE__LOAD_EXAMPLES"] = "True"
Review Comment:
I'd rather start env from scratch for every command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r849314271
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ return
+
+ print(f"\nWiping and recreating {airflow_home_dir}")
+
+ if not airflow_home_dir == airflow_sources:
+ print("AIRFLOW_HOME and Source code are in the same path")
+ print(
+ f"When running this script it will delete all files in path {airflow_home_dir} "
+ "to clear dynamic files like config/logs/db"
+ )
+ print("Please move the airflow source code elsewhere to avoid deletion")
+ return
Review Comment:
```suggestion
sys.exit(3)
```
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
Review Comment:
We should check where the sources are by `Path(__file__).parents[x]` (x= 4 I think?)) this is used elsewhere in the code and makes sure that it does not matter what was the cwd() when the script was run (for example if someone runs it directly from `./script/tools` it will continue to work,
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
Review Comment:
```suggestion
sys.exit(1)
```
If someone automates the script this will indicate that there was an error. Additionally error code might indicate the reason for error.
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
Review Comment:
You can also check here for "if not __main__ " and rais an error if someone tries to accidentally import it.
This script should only be run as "main" script.
There are a few python script in scripts/ci/pre-commit that do similar check
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ return
+
+ print(f"\nWiping and recreating {airflow_home_dir}")
+
+ if not airflow_home_dir == airflow_sources:
+ print("AIRFLOW_HOME and Source code are in the same path")
+ print(
+ f"When running this script it will delete all files in path {airflow_home_dir} "
+ "to clear dynamic files like config/logs/db"
+ )
+ print("Please move the airflow source code elsewhere to avoid deletion")
+ return
+
+ clean_up_airflow_sources(airflow_home_dir)
+
+ print("\nInstalling requirements...")
+ return_code = pip_install_requirements()
+
+ if return_code != 0:
+ print("Error installing the requirements")
+ print("Try running the command below and rerun virtualenv installation\n")
+ os_type = sys.platform
+ if os_type == "darwin":
+ print("brew install sqlite mysql postgresql openssl")
+ print("export LDFLAGS=\"-L/usr/local/opt/openssl/lib\"")
+ print("export CPPFLAGS=\"-I/usr/local/opt/openssl/include\"")
+ else:
+ print(
+ "sudo apt install build-essential python3-dev libsqlite3-dev openssl"
+ "sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql"
+ )
+ return
+
+ print("\nResetting AIRFLOW sqlite database...")
+ os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False"
Review Comment:
It's better to do:
```
env = os.eniron.copy()
env[] =
subprocess.run(...., env=env).
```
This way there is no accidenta influence of previous commands on the others
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ return
+
+ print(f"\nWiping and recreating {airflow_home_dir}")
+
+ if not airflow_home_dir == airflow_sources:
+ print("AIRFLOW_HOME and Source code are in the same path")
+ print(
+ f"When running this script it will delete all files in path {airflow_home_dir} "
+ "to clear dynamic files like config/logs/db"
+ )
+ print("Please move the airflow source code elsewhere to avoid deletion")
+ return
+
+ clean_up_airflow_sources(airflow_home_dir)
+
+ print("\nInstalling requirements...")
+ return_code = pip_install_requirements()
+
+ if return_code != 0:
+ print("Error installing the requirements")
+ print("Try running the command below and rerun virtualenv installation\n")
+ os_type = sys.platform
+ if os_type == "darwin":
+ print("brew install sqlite mysql postgresql openssl")
Review Comment:
We should **likely** review those commands. They are pretty old and could have been different now in more modern os-es. But we can do it later.
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
Review Comment:
```suggestion
def clean_up_airflow_home(airflow_home: Path):
```
We want to cleanup AIRFLOW_HOME really, not sources. :)
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ return
+
+ print(f"\nWiping and recreating {airflow_home_dir}")
+
+ if not airflow_home_dir == airflow_sources:
+ print("AIRFLOW_HOME and Source code are in the same path")
+ print(
+ f"When running this script it will delete all files in path {airflow_home_dir} "
+ "to clear dynamic files like config/logs/db"
+ )
+ print("Please move the airflow source code elsewhere to avoid deletion")
+ return
+
+ clean_up_airflow_sources(airflow_home_dir)
+
+ print("\nInstalling requirements...")
+ return_code = pip_install_requirements()
+
+ if return_code != 0:
+ print("Error installing the requirements")
+ print("Try running the command below and rerun virtualenv installation\n")
+ os_type = sys.platform
+ if os_type == "darwin":
+ print("brew install sqlite mysql postgresql openssl")
+ print("export LDFLAGS=\"-L/usr/local/opt/openssl/lib\"")
+ print("export CPPFLAGS=\"-I/usr/local/opt/openssl/include\"")
+ else:
+ print(
+ "sudo apt install build-essential python3-dev libsqlite3-dev openssl"
+ "sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql"
+ )
+ return
Review Comment:
```suggestion
sys.exit(4)
```
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ return
Review Comment:
```suggestion
sys.exit(2)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joppevos closed pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
joppevos closed pull request #22971: add script to initialise virtualenv
URL: https://github.com/apache/airflow/pull/22971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #22971:
URL: https://github.com/apache/airflow/pull/22971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #22971:
URL: https://github.com/apache/airflow/pull/22971#issuecomment-1097876129
Also in your commit message
Fixes: #NUMBER
Rather than
Issue: #NUMBER
It will then automatically close the issue when the PR is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850292186
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+if __name__ not in ("__main__", "__mp_main__"):
+ raise SystemExit(
+ "This file is intended to be executed as an executable program. You cannot use it as a module."
+ f"To run this script, run the ./{__file__} command"
+ )
+
+
+def clean_up_airflow_home(airflow_home: Path):
+ if airflow_home.exists():
+ print(f"Removing {airflow_home}")
+ shutil.rmtree(airflow_home, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path(__file__).parents[4] / "airflow")
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ sys.exit(1)
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ sys.exit(2)
+
+ print(f"\nWiping and recreating {airflow_home_dir}")
+
+ if airflow_home_dir == airflow_sources:
+ print("AIRFLOW_HOME and Source code are in the same path")
+ print(
+ f"When running this script it will delete all files in path {airflow_home_dir} "
+ "to clear dynamic files like config/logs/db"
+ )
+ print("Please move the airflow source code elsewhere to avoid deletion")
+
+ sys.exit(3)
+
+ clean_up_airflow_home(airflow_home_dir)
+
+ print("\nInstalling requirements...")
+ return_code = pip_install_requirements()
+
+ if return_code != 0:
+ print("Error installing the requirements")
+ print("Try running the command below and rerun virtualenv installation\n")
Review Comment:
When we add configurable extra, we should extend the message a bit. Something like:
```
This command can fail when you install it with the default "devel-all" extras. But if you need to quickly install airlfow venve and work on a subset of Airflow code it might be enough to install basic "devel" requirements.
You can do it via:
./scripts/tools/initialize_virtualenv.py "devel"
You can also install extras for only small subset of extras, for example:
./scripts/tools/initialize_virtualenv.py "devel,google"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joppevos commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
joppevos commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850153239
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ return
+
+ print(f"\nWiping and recreating {airflow_home_dir}")
+
+ if not airflow_home_dir == airflow_sources:
+ print("AIRFLOW_HOME and Source code are in the same path")
+ print(
+ f"When running this script it will delete all files in path {airflow_home_dir} "
+ "to clear dynamic files like config/logs/db"
+ )
+ print("Please move the airflow source code elsewhere to avoid deletion")
+ return
+
+ clean_up_airflow_sources(airflow_home_dir)
+
+ print("\nInstalling requirements...")
+ return_code = pip_install_requirements()
+
+ if return_code != 0:
+ print("Error installing the requirements")
+ print("Try running the command below and rerun virtualenv installation\n")
+ os_type = sys.platform
+ if os_type == "darwin":
+ print("brew install sqlite mysql postgresql openssl")
Review Comment:
Good point. We can address it outside of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22971:
URL: https://github.com/apache/airflow/pull/22971#issuecomment-1101187318
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r850289330
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+if __name__ not in ("__main__", "__mp_main__"):
+ raise SystemExit(
+ "This file is intended to be executed as an executable program. You cannot use it as a module."
+ f"To run this script, run the ./{__file__} command"
+ )
+
+
+def clean_up_airflow_home(airflow_home: Path):
+ if airflow_home.exists():
+ print(f"Removing {airflow_home}")
+ shutil.rmtree(airflow_home, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
Review Comment:
Actuallly it woudl be great if you can make the "extra" configurable. The "devel_all" extra might be not installable in all circumstances (some libraries might be missing). So while it should be the default, we should be able to give the user opportunity to run it with a different extra or set of those.
So there should be an extra parameter that you can add
./scripts/tools/initialize_virtualenv.py "devel"
or
./scripts/tools/initialize_virtualenv.py "devel,google"
That should enable the user to initialize the venv with other extras.
And whenever we have an error while installing, we shoudl inform the user about it. See the other comment:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #22971: add script to initialise virtualenv
URL: https://github.com/apache/airflow/pull/22971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joppevos commented on a diff in pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
joppevos commented on code in PR #22971:
URL: https://github.com/apache/airflow/pull/22971#discussion_r849311338
##########
scripts/tools/initialize_virtualenv.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def clean_up_airflow_sources(airflow_sources: Path):
+ if airflow_sources.exists():
+ print(f"Removing {airflow_sources}")
+ shutil.rmtree(airflow_sources, ignore_errors=True)
+
+
+def check_if_in_virtualenv() -> bool:
+ return hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
+
+
+def pip_install_requirements() -> int:
+ """
+ install the requirements of the current python version.
+ return 0 if success, anything else is an error.
+ """
+ version = get_python_version()
+ constraint = (
+ f"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-{version}.txt"
+ )
+ pip_install_command = ["pip", "install", "-e", ".[devel-all]", "--constraint", constraint]
+ e = subprocess.run(pip_install_command)
+ return e.returncode
+
+
+def get_python_version() -> str:
+ """
+ return the version of python we are running.
+ """
+ major = sys.version_info[0]
+ minor = sys.version_info[1]
+ return f"{major}.{minor}"
+
+
+def main():
+ """
+ Setup local virtual environment.
+ """
+ airflow_home_dir = os.environ.get("AIRFLOW_HOME", Path.home() / "airflow") # TODO: change to airflow
+ airflow_sources = str(Path.cwd())
+ if not check_if_in_virtualenv():
+ print(
+ "Local virtual environment not activated.\nPlease create and activate it "
+ "first. (for example using 'python3 -m venv venv && source venv/bin/activate')"
+ )
+ return
+
+ print("Initializing environment...")
+ print(f"This will remove the folder {airflow_home_dir} and reset all the databases!")
+ response = input("Are you sure? (y/N/q)")
+ if response != "y":
+ return
+
+ print(f"\nWiping and recreating {airflow_home_dir}")
+
+ if not airflow_home_dir == airflow_sources:
+ print("AIRFLOW_HOME and Source code are in the same path")
+ print(
+ f"When running this script it will delete all files in path {airflow_home_dir} "
+ "to clear dynamic files like config/logs/db"
+ )
+ print("Please move the airflow source code elsewhere to avoid deletion")
+ return
+
+ clean_up_airflow_sources(airflow_home_dir)
+
+ print("\nInstalling requirements...")
+ return_code = pip_install_requirements()
+
+ if return_code != 0:
+ print("Error installing the requirements")
+ print("Try running the command below and rerun virtualenv installation\n")
+ os_type = sys.platform
+ if os_type == "darwin":
+ print("brew install sqlite mysql postgresql openssl")
+ print("export LDFLAGS=\"-L/usr/local/opt/openssl/lib\"")
+ print("export CPPFLAGS=\"-I/usr/local/opt/openssl/include\"")
+ else:
+ print(
+ "sudo apt install build-essential python3-dev libsqlite3-dev openssl"
+ "sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql"
+ )
+ return
+
+ print("\nResetting AIRFLOW sqlite database...")
+ os.environ["AIRFLOW__CORE__LOAD_EXAMPLES"] = "False"
+ os.environ["AIRFLOW__CORE__UNIT_TEST_MODE"] = "False"
+ os.environ["AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED"] = "False"
+ os.environ["AIRFLOW__CORE__DAGS_FOLDER"] = f"{airflow_sources}/empty"
+ os.environ["AIRFLOW__CORE__PLUGINS_FOLDER"] = f"{airflow_sources}/empty"
+ subprocess.run(["airflow", "db", "reset", "--yes"])
Review Comment:
This part is migrated from breeze legacy (which is in Bash). It does not feel so Pythonic now with the subprocesses and the env variables. I am open for suggestions here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joppevos commented on pull request #22971: add script to initialise virtualenv
Posted by GitBox <gi...@apache.org>.
joppevos commented on PR #22971:
URL: https://github.com/apache/airflow/pull/22971#issuecomment-1099099140
> Few more comments. I tested the script and after few fixes it works as intended.
>
> I have a mall improvement proposal with making extra configurable - that will be really useful for some users!
Thank you for the quick review! 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org