You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/28 13:54:58 UTC
[GitHub] [airflow] Bowrna opened a new pull request #21145: enter the shell breeze2 environment
Bowrna opened a new pull request #21145:
URL: https://github.com/apache/airflow/pull/21145
closes: #20960
related: #20960
Breeze2 - enter the shell in breeze2
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r805295828
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
@potiuk Do you think it's better to log all the env variable used during this docker-compose command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1033478451
@potiuk I tried running build-ci-image command with Breeze2 and I get the following error. Is there any change that I missed to incorporate ?
```
(airflow-env) sathishkannan@Sathishs-MacBook-Air airflow % ./Breeze2 build-ci-image
[+] Building 0.8s (3/3) FINISHED
=> [internal] load build definition from Dockerfile.ci 0.0s
=> => transferring dockerfile: 41B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> ERROR [internal] load metadata for ghcr.io/apache/airflow/main/python:3.7-slim-buster 0.7s
------
> [internal] load metadata for ghcr.io/apache/airflow/main/python:3.7-slim-buster:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 403 Forbidden
========================= OUTPUT start ============================
None
None
========================= OUTPUT end ============================
Traceback (most recent call last):
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/bin/Breeze2", line 33, in <module>
sys.exit(load_entry_point('apache-airflow-breeze', 'console_scripts', 'Breeze2')())
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/breeze.py", line 245, in build_ci_image
build_image(
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/ci/build_image.py", line 96, in build_image
output = run_command(cmd, verbose=verbose, text=True)
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/utils/run_utils.py", line 45, in run_command
return subprocess.run(cmd, check=check, **kwargs)
File "/Users/sathishkannan/.pyenv/versions/3.8.10/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker', 'build', '--build-arg', 'PYTHON_BASE_IMAGE=ghcr.io/apache/airflow/main/python:3.7-slim-buster', '--build-arg', 'AIRFLOW_VERSION=2.3.0.dev0', '--build-arg', 'AIRFLOW_BRANCH=main', '--build-arg', 'AIRFLOW_EXTRAS=devel_ci', '--build-arg', 'AIRFLOW_PRE_CACHED_PIP_PACKAGES=true', '--build-arg', 'ADDITIONAL_AIRFLOW_EXTRAS=', '--build-arg', 'ADDITIONAL_PYTHON_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_COMMAND=', '--build-arg', 'ADDITIONAL_DEV_APT_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_ENV=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_COMMAND=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_DEPS=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_ENV=', '--build-arg', 'UPGRADE_TO_NEWER_DEPENDENCIES=false', '--build-arg', 'CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow', '--build-arg', 'AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main', '--build-arg', 'AIRFLOW_CONSTRAINTS=constraints-source-providers', '--build-arg', 'AIRFLOW_IMAGE_REPOSITORY=https://github.co
m/apache/airflow', '--build-arg', 'AIRFLOW_IMAGE_DATE_CREATED=2022-02-09T13:50:13Z', '--build-arg', 'BUILD_ID=0', '--build-arg', 'COMMIT_SHA=ebf760871470446fb5e0cd58edf8fec32547e65e', '--cache-from', 'ghcr.io/apache/airflow/main/ci/python3.7:latest', '-t', 'ghcr.io/apache/airflow/main/ci/python3.7:latest', '--target', 'main', '.', '-f', '/Users/sathishkannan/code/airflow/Dockerfile.ci']' returned non-zero exit status 1.
(airflow-env) sathishkannan@Sathishs-MacBook-Air airflow %
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808957232
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
Review comment:
Hey @Bowrna - here are few examples of tests using mocking approach. Actually it was alredy useful to add the tests because I actually FOUND two small issues when I added them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1033705376
> Most flags are applicable to the shell command as it will run build when needed.
Just take all the same flags as `build-ci-image` takes + `--github-image-id` . That will be good start. We can always add the missing ones later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1043043876
Doing a lot of builds tend to accumulate a lot of "garbage" for docker and "prune" is the way to clean it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038210557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1041254478
> > @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
>
> Execute the command. The flow should be like that:
>
> 1. User runs ./breeze without `--force-build`:
>
> * if the image is "fresh" (important files have not changed) - just start breeze shell without asking any questions
> * if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
> y -> build (and the user continues to shell after it is finished).
> n -> the user continues to shell without building
> q -> quit
>
> 2. User runs ./breeze with `--force-build`:
>
> * no matter what the "build is started" first (and the user continues to shell after it is finished).
>
> There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
>
> If the user is not rebased, then we print the warning "This might take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
@potiuk I have added code to support this part. But i didn't add the timeout part yet. After timeout, how do we need to proceed? Do we need to take that as yes/no from user? Could you point me to the code that checks if the user is rebased to latest "main"? That part is not added too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039033267
I think the easiest aand most pythonic way is described as accepted answer here (bit manipulation_:
https://stackoverflow.com/questions/25988471/remove-particular-permission-using-os-chmod
For files:
```
current = stat.S_IMODE(os.stat(file).st_mode)
new = current & ~stat.S_IWGRP & ~stat.SIWOTH # Removes group/other write permission
os.chmod(file, new)
```
For directories:
```
current = stat.S_IMODE(os.stat(dir).st_mode)
new = current & ~stat.S_IWGRP & ~stat.S_IWOTH # Removes group/other write permission
new = new | stat.S_IXGRP | stat.S_IXOTH # Add group/other execute permission (to be able to list directories)
os.chmod(dir, new)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024939890
> "trap" makes sure that no matter if we nicely exist or Ctrl-C, or fail, those files are automatically deleted. The SKIP_CACHE_DELETION variable was just an escape hatch to not delete the files while debugging. I think we do not need it any more (and we can always coment out the code that deletes it when debugging).
>
> In Python this "trap" functionality (not the same but good enough) can be achieved by https://docs.python.org/3/library/atexit.html. And I think this is the right way.
Oh wow :) I am learning a lot of new things from this one PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025659012
You can read more about the bash magic you can do with variable substitution (this is the exact thing) https://tldp.org/LDP/abs/html/parameter-substitution.html
Also what's part of the magic here:
* `${1//[.-]/ }` - the 'space' between `/` and `}` matter - this is how we get the parameters to be separated - basically we replace any "." or "-" with space with this construct.
* It is important that there is no `"` around it. The `printf "%03d%03d%03d%.0s" "${1//[.-]/ }"` would not work because the "2 20 1" would be seen as single parameter. This is kinda unusual use and usually you want those quotes. This is why shellcheck (our pre-commit static check for bash code) complained and this is why I had to add ` # shellcheck disable=SC2086,SC2183 ` to stop it from complaining.
Now I think you should know why we want to get rid of Bash :). If explaining one line of bash takes 2-3 paragraph of text, it means that we should get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r815433421
##########
File path: dev/breeze/src/airflow_breeze/breeze.py
##########
@@ -75,19 +111,78 @@ def version():
@option_verbose
-@main.command()
-def shell(verbose: bool):
+@main.command(
+ context_settings=dict(
+ ignore_unknown_options=True,
+ allow_extra_args=True,
+ ),
+)
+@option_python_version
+@option_backend
+@click.option('--integration', type=click.Choice(ALLOWED_INTEGRATIONS), multiple=True)
+@click.option('-L', '--build-cache-local', is_flag=True)
+@click.option('-U', '--build-cache-pulled', is_flag=True)
+@click.option('-X', '--build-cache-disabled', is_flag=True)
+@click.option('--production-image', is_flag=True)
Review comment:
Correct. This will be spearate command ("build-prod-image").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1022975440
@potiuk i can see several env variables in `scripts/ci/libraries/_initialization.sh`. When the breeze is in bash, this can be set as env variable and can be executed. How do you think we can handle this while moving it to python? Do we have to use`os.environ`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054329427
That was a temporary problem with `rich-click` - but already handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1056781637
@potiuk Could you share your reviews about this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054550160
@potiuk I have some issue in resolving conflicts. It shows only one conflict, but when i resolve and push the code, many of the changes that i made goes missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039036939
I will take a look later today and leave my comments. The "production" image part is not really part of this task, so you can leave it for later :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039144587
> @potiuk I am not very clear about this part of the code too. How does this command helps to find the deleted files.
You can do it in python by simply checking if the file exists before attempting to run chmod on it.
This one is "bash-y" way of handling the case that you had a file in git, but you just deleted it in your commit. In this case it will be present in the `git ls-files` output, but it will not be physically there, so running "chmod" will fail :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1035006093
> @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
Execute the command. The flow should be like that:
1) User runs ./breeze without `--force-build`:
* if the image is "fresh" (important files have not changed) - just start breeze shell
* if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
y -> build
n -> continue without building
q -> quit
2) User runs ./breeze with `--force-build`:
* no matter what the "build is started"
There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
If the user is not rebased, then we print the warning "This will take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1034473330
@potiuk Can you tell me if we need to handle the flag `--force-build-images`? Is it still relevant after the docker buildkit part?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1033478451
@potiuk I tried running build-ci-image command with Breeze2 and I get the following error. Is there any change that I missed to incorporate ?
```
(airflow-env) sathishkannan@Sathishs-MacBook-Air airflow % ./Breeze2 build-ci-image
[+] Building 0.8s (3/3) FINISHED
=> [internal] load build definition from Dockerfile.ci 0.0s
=> => transferring dockerfile: 41B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> ERROR [internal] load metadata for ghcr.io/apache/airflow/main/python:3.7-slim-buster 0.7s
------
> [internal] load metadata for ghcr.io/apache/airflow/main/python:3.7-slim-buster:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 403 Forbidden
========================= OUTPUT start ============================
None
None
========================= OUTPUT end ============================
Traceback (most recent call last):
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/bin/Breeze2", line 33, in <module>
sys.exit(load_entry_point('apache-airflow-breeze', 'console_scripts', 'Breeze2')())
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/breeze.py", line 245, in build_ci_image
build_image(
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/ci/build_image.py", line 96, in build_image
output = run_command(cmd, verbose=verbose, text=True)
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/utils/run_utils.py", line 45, in run_command
return subprocess.run(cmd, check=check, **kwargs)
File "/Users/sathishkannan/.pyenv/versions/3.8.10/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker', 'build', '--build-arg', 'PYTHON_BASE_IMAGE=ghcr.io/apache/airflow/main/python:3.7-slim-buster', '--build-arg', 'AIRFLOW_VERSION=2.3.0.dev0', '--build-arg', 'AIRFLOW_BRANCH=main', '--build-arg', 'AIRFLOW_EXTRAS=devel_ci', '--build-arg', 'AIRFLOW_PRE_CACHED_PIP_PACKAGES=true', '--build-arg', 'ADDITIONAL_AIRFLOW_EXTRAS=', '--build-arg', 'ADDITIONAL_PYTHON_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_COMMAND=', '--build-arg', 'ADDITIONAL_DEV_APT_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_ENV=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_COMMAND=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_DEPS=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_ENV=', '--build-arg', 'UPGRADE_TO_NEWER_DEPENDENCIES=false', '--build-arg', 'CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow', '--build-arg', 'AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main', '--build-arg', 'AIRFLOW_CONSTRAINTS=constraints-source-providers', '--build-arg', 'AIRFLOW_IMAGE_REPOSITORY=https://github.co
m/apache/airflow', '--build-arg', 'AIRFLOW_IMAGE_DATE_CREATED=2022-02-09T13:50:13Z', '--build-arg', 'BUILD_ID=0', '--build-arg', 'COMMIT_SHA=ebf760871470446fb5e0cd58edf8fec32547e65e', '--cache-from', 'ghcr.io/apache/airflow/main/ci/python3.7:latest', '-t', 'ghcr.io/apache/airflow/main/ci/python3.7:latest', '--target', 'main', '.', '-f', '/Users/sathishkannan/code/airflow/Dockerfile.ci']' returned non-zero exit status 1.
(airflow-env) sathishkannan@Sathishs-MacBook-Air airflow %
```
Edit: @potiuk I noticed its change of buildkit. I used dry_run_docker to figure out the modified docker command to execute the build image part. I will fix it and update it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1042725652
The issue #21145 hs been fixed already. I test the `./Breeze2 build-ci-image` and it seems to work now @Bowrna . Some more comments follow shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808906269
##########
File path: dev/breeze/src/airflow_breeze/utils/run_utils.py
##########
@@ -40,12 +41,22 @@ def run_command(
verbose: bool = False,
suppress_raise_exception: bool = False,
suppress_console_print: bool = False,
+ env: Optional[Mapping[str, str]] = None,
**kwargs,
):
if verbose:
- console.print(f"[blue]$ {' '.join(shlex.quote(c) for c in cmd)}")
+ command_to_print = ' '.join(shlex.quote(c) for c in cmd)
Review comment:
And Also it allows to update the "verbose" part of run_command - instead of printing the variables and commands outside - I can now also print all the environment variables here - which means that it will work in the future as well in any place where you pass env parameter to run_command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1043042607
> @potiuk I face issues in build-image even after pulling the code. I am not sure if I did anything wrong? i tried in different terminal and still facing error.
I think you might simply have problem with disk space :). Try to run `docker sytem prune` and if it does not help `docker system prune --all` -> the last one cleans up docker entirely so you will have to download a lot to get it back, but sometimes I do it when i have weird problems like that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024884489
> @potiuk Could you share your views about getting host user id and group id? Do we have to find its equivalent in windows to make this work? Also, can you explain to me more about why we need to find the host user id and group id and how it's used? There are a few comments about it in the code, but I couldn't fully understand it.
This is only really needed on Linux. On Windows/MacOS they are not needed and can be empty (I believe - to be checked).
The reason is that on Linux files that are mounted from the host to the container are mounted using native filesystem. This basically means that any file created inside the container will keep the userid /group id that are used in container also in the host.
For example if we have user id 50001 and group id 501 in the container, any file we create in container will keep the same user is and group id in the host. But those user/group ids might not exist in the Host. - if we create a user 50001 in the container, the id will remain like that on the host, when we exit from the container. This is very problematic on linux because when we map "logs" directory and some logs (and directories) are created there, they might be owned by a non-existing user after we exit. And we want to be able to see the logs outside of the container because that's where we usually have IDE and that's where we keep reading those and analyse them.
Then, the problem is that if you want to delete such folders and files, you need to use `sudo` in the host, because your regular user has no access to it. This is big problem especially if files are created inside your source directory (which is also mounted to the container) - for example it will prevent you from switching branches easily because git will not be able to remove some files and it will refuse to switch branches.
There is also "reverse" problem - if you create files in a host with no "all" permissions, and you mount them inside the container, and container runs as "different" user, the user in container cannot access to those file (unless you run as root inside the container - root inside the container is equivalent to root in host and can access and update all files).
This can be mitigated by "user remapping" - https://docs.docker.com/engine/security/userns-remap/ - but this can only be configure at the "docker daemon" level, and this is something we should not require an average user should do, also the problem with user remapping is that it is "global" setting. It will remap your user for all containers and in many cases this is not what you really want.
So In order to avoid that we have a few things:
a) We use `root` user in container - all the files are created and run as root user. This is not recommended for production but it is great for CI - because you can freely create and read any mounted files (no matter what user), you can also run pip/apt etc. without sudo and it is generally much more convenient for many development tasks. The side effect of that is that all files created in the container have root user/group set.
b) we pass HOST_USER_ID and HOST_GROUP_ID to the container, so that we know who is the user on the host. Depending on lthe linux distro and even depending on your configuration (how many users you have created and in which sequence) - the UID can be different.
c) when the user enters the container, we set a "trap"L `add_trap "in_container_fix_ownership" EXIT HUP INT TERM` - this trap runs "fix_ownership" script that looks for all created files in the directories where we expect we will create files:
```
"/files"
"/root/.aws"
"/root/.azure"
"/root/.config/gcloud"
"/root/.docker"
"/opt/airflow/logs"
"/opt/airflow/docs"
"/opt/airflow/dags"
"${AIRFLOW_SOURCES}"
```
Whenever we exit, or terminate the container, this script is executed and it finds all files owned by "root" in those directories and changes their ownership to be HOST_USER/HOST_GROUP. This way when you exit the containers on linux, the files are owned by the host user, and can be easily deleted - either manually or when you change branches.
On MacOS and Windows this is not needed. Both MacOS and Windows use "user-space" filesystems to mount files. The filestystems are far slower than the native filesystem (many times actually) - which impacts the speed of runnign airlfow in docker container on MacOS and Windows. However they automatically remap the user - all the files created inside the containers are automatically remapped to have the "host" user ownership and there is no need to fix the ownership for those cases.
I hope it is clearer now. I will create an ADR out of that comment :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025659012
You can read more about the bash magic you can do with variable substitution (this is the exact thing) https://tldp.org/LDP/abs/html/parameter-substitution.html
Also what's part of the magic here:
* `${1//[.-]/ }` - the 'space' between `/` and `}` matter - this is how we get the parameters to be separated
* It is important that there is no `"` around it. The `printf "%03d%03d%03d%.0s" "${1//[.-]/ }"` would not work because the "2 20 1" would be seen as single parameter. This is kinda unusual use and usually you want those quotes. This is why shellcheck (our pre-commit static check for bash code) complained and this is why I had to add ` # shellcheck disable=SC2086,SC2183 ` to stop it from complaining.
Now I think you should know why we want to get rid of Bash :). If explaining one line of bash takes 2-3 paragraph of text, it means that we should get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1026014527
> @potiuk I was testing with one of the versions of docker that is lower than the minimum docker version
>
> ```
> docker version --format '{{.Client.Version}}'
> 19.03.5
> ```
>
> But when validated with semver library, it says it's a invalid version name.
>
> ```
> In [2]: VersionInfo.isvalid("19.03.5")
too strict semver :)? I guess it's the leading 0 in 03. Maybe normalizing it similar to the SO thread would help (or maybe semver has an option to accept leading 0s ?) : https://stackoverflow.com/questions/52510965/how-to-remove-leading-zeros-from-version-number-string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025087893
@potiuk I am working on checking the docker version code.
https://github.com/apache/airflow/blob/1d4b709e20b07c6f0b5d1bab1935e19557df2913/scripts/ci/libraries/_initialization.sh#L922-L958
Do we have to use the bash commands you have used in lines 924 and 935 via subcommand in python? Or do we have to try a different way? Also, I don't understand the comment added `In GitHub Code QL, the version of docker has +azure suffix which we should remove`. Could you explain to me that part?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025089492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1029581924
> That's simple. The global constants should contain the variables that do not change when parameters change. The dataclass should be used when you have a "calculated" value which depends on the parameters that you passed to the "breeze" command.
> Actually that was one of the difficulty with Bash - because you can hardly distinguish those - env variable is an env variable and whether it is "unchangeable" or whether it has been calculated on-the-fly using passed parameters it is hard to tell. The Dataclass is our chance to distinguish between those two cases.
yes, I could understand this part, but figuring out if the variable remains constant / varies depending on 'parameters' passed to breeze command is the part where I am confused as you mentioned. Say example `USE_AIRFLOW_VERSION ` does it have to go to global constants or dataclass. Does the user can set this value / does it have to remain constant? How do I figure out that? @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1022975440
@potiuk i can see several env variables in `scripts/ci/libraries/_initialization.sh`. When breeze is in bash, this can be set as env variable and can be executed. How do you think we can handle this while moving it to python?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1023245481
> > @potiuk i can see several env variables in scripts/ci/libraries/_initialization.sh.
>
> We definitely should not try to set all of those. They are needed by different parts of "old breeze". The variables that we should pass are exactly those which are "generated" in the "dc_ci" script - just take a look what you have in `./build/dc_ci` and those are the only environment variables that are really needed to run the CI. Another place (we have a few of those) that you can reconcile are in `scripts/ci/docker-compose/_docker.env` and the same variables are repeated in `scripts/ci/docker-compose/base.yaml` - maybe there are some slight differences we will have to reconcile.
>
> BTW. The reason why we have it in many places is that there are some difference in how "docker-compose 1" and "docker-compose 2" work. This will likely go away soon (Docker-compose 2 has been introduced recently and I think we will soon be able to deprecate docker-compose 1) at which time we will be able to remove one of those lists.
>
> I think (can't remember) I have a pre-commit that keeps those in-check - but maybe I have not done that, so better to check and reconcile it now.
>
> Regardless-having yet another list of the variables in python in "global_constants" that will be the "source of truth" would be great.
@potiuk ok, I will check the vars in dc_ci script. Also you have mentioned about reconciling vars in `scripts/ci/docker-compose/_docker.env` and `scripts/ci/docker-compose/base.yaml`. I will check that part to understand more about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1028939429
@potiuk could you explain to me what this `-n ` check does in this bash code?
https://github.com/apache/airflow/blob/ddb5246bd1576e2ce6abf8c80c3328d7d71a75ce/breeze#L714-L718
Also, I am quite confused about what variables should be in global_constants and what has to be in the dataclass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038178444
> @potiuk Could you explain more about this part? I could understand it fix the permission issue, but I don't understand what problems it is causing and why it has to be fixed this way. Also do we need to implement the same by running the command in subprocess module python?
https://github.com/apache/airflow/blob/cca2f942ea9363303a041f00e0f180924cb840a9/scripts/ci/libraries/_permissions.sh#L19-L53
The problem to solve is very subtle. It is a result of a few combined issues.
1) Git does not store "group write" permissions for files you put in the repository. This is a historical decision also coming from the fact that "group" permissions only make sense for POSIX compliant systems (So not for Windows)
2) When the files are checked out, by default git uses "umask" of the current system to determine whether the group permissions to write are set or not https://git-scm.com/docs/git-config#Documentation/git-config.txt-coresharedRepository (that's the default behaviour when 'false' is set.
3) Unfortunately POSIX does not define what should be the default umask, It usually is either `0022` or `0002` - depending on the system. This basically mean that your file when you check it out with git will be "group read" when umask is `0022` but will be "group read-write" when umask is `0002'. More about umask and difference between those two setttings can be found here: https://www.cyberciti.biz/tips/understanding-linux-unix-umask-value-usage.html
So when you check-out airflow project on `0022` system you will get this:
```
ls -la scripts/ci
total 132
drwxr-xr-x 19 jarek jarek 4096 Feb 5 20:49 .
drwxr-xr-x 8 jarek jarek 4096 Feb 5 20:49 ..
drwxr-xr-x 2 jarek jarek 4096 Feb 5 20:49 build_airflow
```
But when you check-out airflow projecet on `0002` umask system, you will get this: (Note "w" in the group permission part):
```
ls -la scripts/ci
total 132
drwxrwxr-x 19 jarek jarek 4096 Feb 5 20:49 .
drwxrwxr-x 8 jarek jarek 4096 Feb 5 20:49 ..
drwxrwxr-x 2 jarek jarek 4096 Feb 5 20:49 build_airflow
```
I think pretty much all Linux distributions use `0022` umask for root. But when it comes to "regular" users, it is different - Ubuntu up until recently used "0002" for regular users, and Debian/Mint "0022". The result is - we cannot rely on the "w" bit being set for files when users checks it out - it might or might not be set and it's beyond of our control when `git checkout` is executed.
On it's own - this is not a problem, but it is a HUGE problem when it comes to caching Docker builds. The problem is, that when you build Docker image, docker actually uses the permissions on the file to determine whether a file changed or not. In the case above. those two "build_airflow" files will have identical content but Docker finds them "different" (becaus of the "w" permission).
In this case whenever in Dockerfile we will use:
```
COPY scripts/ci/build_airlfow /opt/airflow/scripts/ci/build
```
Docker "thinks" that the file has changed and will simply invalidate the cache. So when we use remote cache (as we do) this means that docker will always rebuild the docker image from that step, because it will not realise that this is in fact the same file. This is a huge problem in our case, because in order to speed up local rebuilds of Breeze image. we use remote cache and we have to make sure that the file that was used to build airflow will be properly seen as "unchanged" by anyone who builds the image locally - in order to heavily optimize the build time (pulling cached layers is usually much faster than rebuilding them - especially that rebuild often involves pulling `pip` packages and `apt` packages.
That's why we fix the problem just before building. We basically find all files and directories which are present in git repository and that have "w" bit set and remove the bit. After this operation none of the files that are going to be used for docker build will have the "w" bit set for them - and the cache is invalidated.
And yeah. We have to do equivalent of this in Python. Submodule is one way but we could also work out similar approach using Pyhon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039144587
> @potiuk I am not very clear about this part of the code too. How does this command helps to find the deleted files.
You can do it in python by simply checking if the file exists before attempting to run chmod.
This one is "bash-y" way of handling the case that you had a file in git, but you just deleted it in your commit. In this case it will be present in the `git ls-files` output, but it will not be physically there so running "chmod" will fail :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039144587
> @potiuk I am not very clear about this part of the code too. How does this command helps to find the deleted files.
You can do it in python by simply checking if the file exists before attempting to run chmod on it.
This one is "bash-y" way of handling the case that you had a file in git, but you just deleted it in your commit. In this case it will be present in the `git ls-files` output, but it will not be physically there so running "chmod" will fail :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039063349
@potiuk I am not very clear about this part of the code too. How does this command helps to find the deleted files.
https://github.com/apache/airflow/blob/28378d867afaac497529bd2e1d2c878edf66f460/scripts/ci/libraries/_permissions.sh#L35-L40
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r812673596
##########
File path: dev/breeze/tests/test_cache.py
##########
@@ -62,6 +67,17 @@ def test_read_from_cache_file(param):
if param_value is None:
assert None is param_value
else:
- allowed, param_list = check_if_values_allowed(param + 's', param_value)
+ allowed, param_list = check_if_values_allowed(param, param_value)
if allowed:
assert param_value in param_list
+
+
Review comment:
@potiuk Could you share your views on this test that I tried?
In the below code, If i tried `mock_path.assert_called_with(AIRFLOW_SOURCES / ".build"/ param)` it didn't work. Can you explain why it doesn't work? I tried to find myself. But that part is not very clear to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054276449
@potiuk I have added timeout and included a few tests too. Did I miss anything else in this PR? You could run enter shell command and share your reviews :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054306068
> smth strange failed in build images. Let me close/reopen to check.
Yes, I noticed that and thought of asking you. Ping me if i can help in fixing it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054424725
@Jarek Potiuk I have added some dummy unused variables to test. cleaning it up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1042671650
@potiuk I am currently working on writing tests and there are many functions that execute the docker command via subprocess. How can we add tests for these functions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808902701
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -166,16 +162,14 @@ def build_image_checks(verbose: bool, shell_params: ShellBuilder):
check_docker_resources(verbose, str(shell_params.airflow_sources), shell_params.airflow_ci_image_name)
cmd = ['docker-compose', 'run', '--service-ports', '--rm', 'airflow']
cmd_added = shell_params.command_passed
+ env_variables = construct_env_variables_docker_compose_command(shell_params)
if cmd_added is not None:
cmd.extend(['-c', cmd_added])
if verbose:
- shell_params.print_badge_info
- show_cmd = construct_arguments_docker_compose_command(shell_params)
- show_cmd.extend(cmd)
- console.print('Command to run:', (' ').join(show_cmd))
- output = run_command(cmd, verbose=False, text=True)
+ shell_params.print_badge_info()
Review comment:
Here small correction - print_badge_info is a function not property. I corrected it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808902167
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -166,16 +162,14 @@ def build_image_checks(verbose: bool, shell_params: ShellBuilder):
check_docker_resources(verbose, str(shell_params.airflow_sources), shell_params.airflow_ci_image_name)
cmd = ['docker-compose', 'run', '--service-ports', '--rm', 'airflow']
cmd_added = shell_params.command_passed
+ env_variables = construct_env_variables_docker_compose_command(shell_params)
Review comment:
Now - I store the env vars in the local variable here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808904730
##########
File path: dev/breeze/src/airflow_breeze/utils/run_utils.py
##########
@@ -40,12 +41,22 @@ def run_command(
verbose: bool = False,
suppress_raise_exception: bool = False,
suppress_console_print: bool = False,
+ env: Optional[Mapping[str, str]] = None,
Review comment:
This is the "gist" and the problem you had been struggling with. Instead of modifying "os.environ" I pass the variables as dictionary to the "run_command" - this way I only modify the variables for the command that I run, but I leave the env variables intact for the current process. This is much "cleaner" way of doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1042811453
This actually looks great @Bowrna ! I had finally time to take a closer look and try it out and it "ALMOST" worked from the first try - I added the (rather small) fixup to the problem you struggled with when passing the env variables - mostly to show you quickly how this could be done better. I added all the explanation so that you can see my reasoning and why this way might be better :).
I will also add another small fixup with example test in a moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808959780
##########
File path: dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
##########
@@ -116,7 +116,7 @@ def check_docker_permission(verbose) -> bool:
def compare_version(current_version: str, min_version: str) -> bool:
- return version.parse(current_version) > version.parse(min_version)
+ return version.parse(current_version) >= version.parse(min_version)
Review comment:
This is the first error I found (the 20.10.0 version was actually failing the check :scream: )
##########
File path: dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
##########
@@ -145,7 +145,7 @@ def check_docker_version(verbose: bool):
else:
console.print(
f'Your version of docker is too old:{docker_version}. Please upgrade to \
- at least ${MIN_DOCKER_VERSION}'
+ at least {MIN_DOCKER_VERSION}'
Review comment:
And this is the second (there was extra '$' printed.
So if you ask me if it is worth to add the tests - yep. It is :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025089492
> Do we have to use the bash commands you have used in lines 924 and 935 via subcommand in python? Or do we have to try a different way? Also, I don't understand the comment added `In GitHub Code QL, the version of docker has +azure suffix which we should remove`. Could you explain to me that part?
Yeah. We should run the `docker commands via subprocess`. We **could** potentially use docker API in python, but running docker as `docker` command is precisely what we want to check . In this check we want to check if the user has:
* good docker command line version and it is on the PATH
* the version is properly configured - when you install docker on Linux you have to perform [additional steps](https://docs.docker.com/engine/install/linux-postinstall/) to make docker available for you user - the "permission" error is the way to check it (however it would be great to see if we have the message right, it could have changed). The way to check it is to remove yourself from docker group or run docker as a diferent user who is not in `docker` group
The best way to check it is to actually run the commands :)
> In GitHub Code QL, the version of docker has +azure suffix which we should remove.
The version check simpl checks the min major/minor version and in Python it can be written in a nicer way. The point here is that usualy the version prnted by Docker is like that:
```
docker version --format '{{.Client.Version}}'
20.10.7
```
However on of our users who run ./breeze in Azure Code QL ( which is Azure's CI system) got error that the version could not be parsed, because in Azure Code QL, they MODIFIED the version of docker they have installed and the version returened was something like:
```
docker version --format '{{.Client.Version}}'
20.10.7+azure
```
:scream: :scream: :scream: :scream: :scream:
So our version check must include also this case - parsing the version .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1029581924
> That's simple. The global constants should contain the variables that do not change when parameters change. The dataclass should be used when you have a "calculated" value which depends on the parameters that you passed to the "breeze" command.
yes, I could understand this part, but figuring out if the variable remains constant / varies depending on 'parameters' passed to breeze command is the part where I am confused. Say example `USE_AIRFLOW_VERSION ` does it have to go to global constants or dataclass. Does user can set this value / it has to remain constant? How do i figure out that? @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1026014527
> @potiuk I was testing with one of the versions of docker that is lower than the minimum docker version
>
> ```
> docker version --format '{{.Client.Version}}'
> 19.03.5
> ```
>
> But when validated with semver library, it says it's a invalid version name.
>
> ```
> In [2]: VersionInfo.isvalid("19.03.5")
too strict semver :)? I guess it's the leading 0 in 03. Maybe normalizing it similar to would help (or maybe semver has an option to accept leading 0s ?) : https://stackoverflow.com/questions/52510965/how-to-remove-leading-zeros-from-version-number-string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025636718
@potiuk Can you explain to me what's happening in this code part? Couldn't decipher what this regex part is doing?
https://github.com/apache/airflow/blob/1d4b709e20b07c6f0b5d1bab1935e19557df2913/scripts/ci/libraries/_initialization.sh#L916-L920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1027628650
@potiuk Here we have appended `backend_port_docker_compose_file` var in both `MOUNT_SELECTED_LOCAL_SOURCES` and `MOUNT_ALL_LOCAL_SOURCES` check. Could you explain why it's done that way? Could we move that out of check?
https://github.com/apache/airflow/blob/d353f023ff8856c00b9f054526cb2e40ff0116ae/breeze#L702-L718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1026059505
> @potiuk Could we check this one for version comparison? https://packaging.pypa.io/en/latest/version.html#packaging.version.parse
Sure :). Whatever works and does not require "excessive" dependencies. "Packaging" is done by the same people who maintain "pip" (Basically PyPa Python Software Foundation initiative) so it should be perfectly ok to use it if it does better job than semver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #21145:
URL: https://github.com/apache/airflow/pull/21145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1057946277
I think we are good to go !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054378319
@Bowrna - I also added a fixup with complete removal of `self.production_image` :). It was failing locally when it was just commented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r816095542
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import sys
+from pathlib import Path
+from typing import Dict
+
+import click
+from inputimeout import TimeoutOccurred, inputimeout
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.ci.build_image import build_image
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ fix_group_permissions,
+ get_latest_sha,
+ instruct_build_image,
+ instruct_for_setup,
+ is_repo_rebased,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_env_variables_docker_compose_command(shell_params: ShellBuilder) -> Dict[str, str]:
+ env_variables: Dict[str, str] = {}
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ env_variables[param_name] = str(getattr(shell_params, param_value))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ env_variables[constant_param_name] = str(constant_param_value)
+ return env_variables
+
+
+def build_image_if_needed_steps(verbose: bool, shell_params: ShellBuilder):
+ build_needed = md5sum_check_if_build_is_needed(shell_params.md5sum_cache_dir, shell_params.the_image_type)
+ build_needed = True
+ if build_needed:
+ try:
+ user_status = inputimeout(
+ prompt='\nDo you want to build image?Press y/n/q in 5 seconds\n',
Review comment:
Could i check for coloring text with options other than rich console @potiuk ? As I am using `inputimeout` and `click.confirm` for questions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1030120241
> > yes, I could understand this part, but figuring out if the variable remains constant / varies depending on 'parameters' passed to breeze command is the part where I am confused as you mentioned. Say example `USE_AIRFLOW_VERSION ` does it have to go to global constants or dataclass. Does the user can set this value / does it have to remain constant? How do I figure out that? @potiuk
>
> It's passed by parameter (`--use-airflow-version` in this case) , so it should be in dataclass. You should not be able to change a constants with the command line parameter really :)
Cool :) I didn't notice the parameter. I will remember this way that parameters are user-modifiable and its better to place them in dataclass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1034833059
@potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039033267
I think the easiest aand most pythonic way is described as accepted answer here (bit manipulation):
https://stackoverflow.com/questions/25988471/remove-particular-permission-using-os-chmod
For files:
```
current = stat.S_IMODE(os.stat(file).st_mode)
new = current & ~stat.S_IWGRP & ~stat.S_IWOTH # Removes group/other write permission
os.chmod(file, new)
```
For directories:
```
current = stat.S_IMODE(os.stat(dir).st_mode)
new = current & ~stat.S_IWGRP & ~stat.S_IWOTH # Removes group/other write permission
new = new | stat.S_IXGRP | stat.S_IXOTH # Add group/other execute permission (to be able to list directories)
os.chmod(dir, new)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038210557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808909739
##########
File path: dev/breeze/src/airflow_breeze/utils/run_utils.py
##########
@@ -40,12 +41,22 @@ def run_command(
verbose: bool = False,
suppress_raise_exception: bool = False,
suppress_console_print: bool = False,
+ env: Optional[Mapping[str, str]] = None,
**kwargs,
):
if verbose:
- console.print(f"[blue]$ {' '.join(shlex.quote(c) for c in cmd)}")
+ command_to_print = ' '.join(shlex.quote(c) for c in cmd)
+ # if we pass environment variables to execute, then
+ env_to_print = ' '.join(f'{key}="{val}"' for (key, val) in env.items()) if env else ''
+ # Soft wrap allows to copy&paste and run resulting output as it has no hard EOL
+ console.print(f"[blue]{env_to_print} {command_to_print}[/]", soft_wrap=True)
try:
- return subprocess.run(cmd, check=check, **kwargs)
+ # copy existing environment variables
+ cmd_env = deepcopy(os.environ)
Review comment:
This is also needed here because we need to "add" our environment variables to the existing ones. When you pass "env" to "subprocess.run" - the env passed will replace the current ones (and for example PATH will be missing, so "docker-compose" command will not be found. By cloning the os.environ first and updating the clone with the new env variables, I make sure that the old variables remain and the new ones are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r805176319
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
The problem is that `docker-compose up` does not handle `-e` flag :(
See here:
https://stackoverflow.com/questions/49293967/how-to-pass-environment-variable-to-docker-compose-up
I think you should continue passing them via os.environ (this is basically what is suggested in the last answer as second option.
However I think that requires a bit of modification of the "--verbose" run_command behaviour - you shoudl be able to pass it the list of environment variables to set and the "verbose" information should include it in the command as evn variable setting, so that you can copy&paste it from there.
For example if you run docker-compose with "VAR1"="VAL1" variable, the verbose command should look like:
```
VAR1=VAL1 docker-compose up ......
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r805176319
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
The problem is that `docker-compose up` does not handle `-e` flag :(
See here:
https://stackoverflow.com/questions/49293967/how-to-pass-environment-variable-to-docker-compose-up
I think you should continue passing them via os.environ (this is basically what is suggested in the last answer as second option.
However I think that requires a bit of modification of the "--verbose" run_command behaviour - you shoudl be able to pass it the list of environment variables to set and the "verbose" information should include it in the command as evn variable setting, so that you can copy&paste it from there.
For example if you run docker-compose with "VAR1"="VAL1" variable, the verbose command should look like:
```
VAR1=VAL1 docker-compose up ......
```
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
I think so. The idea for the "verbose" part is to give the user a way to copy&paste the command and re-run it easily (maybe with small modifications) and to be able to diagnose potential bugs in setting the configuration in the CI.
So logging all variables there (i.e. being able to reproduce the exact command that caused the problem) is very important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1037259920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r811287573
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
+ mock_console.print.assert_called_with('See https://docs.docker.com/compose/install/ for instructions. Make sure docker-compose you install is first on the PATH variable of yours.')
+
+
Review comment:
The below function `test_check_docker_compose_version_ok ` is the one that is failing @potiuk
I am not sure about where i am going wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r812737258
##########
File path: dev/breeze/src/airflow_breeze/utils/run_utils.py
##########
@@ -73,3 +97,110 @@ def check_package_installed(package_name: str) -> bool:
Please install using https://pre-commit.com/#install to continue[/]\n"
)
return is_installed
+
+
+def get_filesystem_type(filepath):
+ root_type = "unknown"
+ for part in psutil.disk_partitions():
+ if part.mountpoint == '/':
+ root_type = part.fstype
+ continue
+ if filepath.startswith(part.mountpoint):
+ return part.fstype
+
+ return root_type
+
+
+def calculate_md5_checksum_for_files(md5sum_cache_dir: Path):
+ not_modified_files = []
+ modified_files = []
+ for calculate_md5_file in FILES_FOR_REBUILD_CHECK:
+ file_to_get_md5 = Path(AIRFLOW_SOURCE, calculate_md5_file)
+ md5_checksum = generate_md5(file_to_get_md5)
+ sub_dir_name = file_to_get_md5.parts[-2]
+ actual_file_name = file_to_get_md5.parts[-1]
+ cache_file_name = Path(md5sum_cache_dir, sub_dir_name + '-' + actual_file_name + '.md5sum')
+ file_content = md5_checksum + ' ' + str(file_to_get_md5) + '\n'
+ is_modified = update_md5checksum_in_cache(file_content, cache_file_name)
+ if is_modified:
+ modified_files.append(calculate_md5_file)
+ else:
+ not_modified_files.append(calculate_md5_file)
+ return modified_files, not_modified_files
+
+
+def md5sum_check_if_build_is_needed(md5sum_cache_dir: Path, the_image_type: str) -> bool:
+ build_needed = False
+ modified_files, not_modified_files = calculate_md5_checksum_for_files(md5sum_cache_dir)
+ if len(modified_files) > 0:
+ console.print('The following files are modified: ', modified_files)
+ console.print(f'Likely {the_image_type} image needs rebuild')
+ build_needed = True
+ else:
+ console.print(
+ f'Docker image build is not needed for {the_image_type} build as no important files are changed!'
+ )
+ return build_needed
+
+
+def instruct_build_image(the_image_type: str, python_version: str):
+ console.print(f'The {the_image_type} image for python version {python_version} may be outdated')
+ console.print('Please run this command at earliest convenience:')
+ if the_image_type == 'CI':
+ console.print(f'./Breeze2 build-ci-image --python {python_version}')
+ else:
+ console.print(f'./Breeze2 build-prod-image --python {python_version}')
+ console.print("If you run it via pre-commit as individual hook, you can run 'pre-commit run build'.")
+
+
+def instruct_for_setup():
+ CMDNAME = 'Breeze2'
+ console.print(f"You can setup autocomplete by running {CMDNAME} setup-autocomplete'")
+ console.print(" You can toggle ascii/cheatsheet by running:")
+ console.print(f" * {CMDNAME} toggle-suppress-cheatsheet")
+ console.print(f" * {CMDNAME} toggle-suppress-asciiart")
+
+
+@contextlib.contextmanager
+def working_directory(source_path: Path):
+ # Equivalent of pushd and popd in bash script.
+ # https://stackoverflow.com/a/42441759/3101838
+ prev_cwd = Path.cwd()
+ os.chdir(source_path)
+ try:
+ yield
+ finally:
+ os.chdir(prev_cwd)
+
+
+def change_file_permission(file_to_fix: Path):
Review comment:
Hmm. It should be possible with mocking `os.stat` and `os.chmod`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r815295778
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,199 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import sys
+from pathlib import Path
+from typing import Dict
+
+import click
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.ci.build_image import build_image
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ fix_group_permissions,
+ instruct_build_image,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_env_variables_docker_compose_command(shell_params: ShellBuilder) -> Dict[str, str]:
+ env_variables: Dict[str, str] = {}
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ env_variables[param_name] = str(getattr(shell_params, param_value))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ env_variables[constant_param_name] = str(constant_param_value)
+ return env_variables
+
+
+def build_image_checks(verbose: bool, shell_params: ShellBuilder):
+ fix_group_permissions()
+ build_ci_image_check_cache = Path(
+ BUILD_CACHE_DIR, shell_params.airflow_branch, f".built_{shell_params.python_version}"
+ )
+ if build_ci_image_check_cache.exists():
+ console.print(f'{shell_params.the_image_type} image already built locally.')
+ else:
+ console.print(f'{shell_params.the_image_type} image not built locally')
+ if not shell_params.force_build:
+ build_needed = md5sum_check_if_build_is_needed(
+ shell_params.md5sum_cache_dir, shell_params.the_image_type
+ )
+ if build_needed:
+ try:
Review comment:
@potiuk Could you tell me what's the best way to add a timeout for input? I have used `click.confirm` for now. But I am not sure about the way in which I can add timed input.
While I checked a few options like python select, signal (it's specific to Unix) and I am not sure if it would be suitable. Your suggestions in this part would be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808897118
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -117,18 +117,17 @@
}
-def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
- args_command = []
+def construct_env_variables_docker_compose_command(shell_params: ShellBuilder) -> Dict[str, str]:
Review comment:
Rather than buildign a command - I simply return dictionary of variables that shoudl be used in the form of "key->value" dict. This will be easier to use when you run commands
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r811713172
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
+ mock_console.print.assert_called_with('See https://docs.docker.com/compose/install/ for instructions. Make sure docker-compose you install is first on the PATH variable of yours.')
+
+
Review comment:
That's the best part !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1042713393
finally looking at it now - sorry for some delays in responding.
> @potiuk I am currently working on writing tests and there are many functions that execute the docker command via subprocess. How can we add tests for these functions?
The best way is to mock the execution of the run_command function and check if the right command is executed. Give me an hours - I will revfew the code and add some example test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808907947
##########
File path: dev/breeze/src/airflow_breeze/utils/run_utils.py
##########
@@ -40,12 +41,22 @@ def run_command(
verbose: bool = False,
suppress_raise_exception: bool = False,
suppress_console_print: bool = False,
+ env: Optional[Mapping[str, str]] = None,
**kwargs,
):
if verbose:
- console.print(f"[blue]$ {' '.join(shlex.quote(c) for c in cmd)}")
+ command_to_print = ' '.join(shlex.quote(c) for c in cmd)
+ # if we pass environment variables to execute, then
+ env_to_print = ' '.join(f'{key}="{val}"' for (key, val) in env.items()) if env else ''
+ # Soft wrap allows to copy&paste and run resulting output as it has no hard EOL
+ console.print(f"[blue]{env_to_print} {command_to_print}[/]", soft_wrap=True)
Review comment:
This one is interesting. Originally rich "hard" wraps the lines in console - so when you copy & paste such output, you get line breaks and if you have long line the copy&paste will fail.
The "soft_wrap" parameter of print, removes the hard line breaks, and now you can simply select the command printed in blue, copy &paste it and it should nicely work :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r815436090
##########
File path: dev/breeze/src/airflow_breeze/breeze.py
##########
@@ -75,19 +111,78 @@ def version():
@option_verbose
-@main.command()
-def shell(verbose: bool):
+@main.command(
+ context_settings=dict(
+ ignore_unknown_options=True,
+ allow_extra_args=True,
+ ),
+)
+@option_python_version
+@option_backend
+@click.option('--integration', type=click.Choice(ALLOWED_INTEGRATIONS), multiple=True)
+@click.option('-L', '--build-cache-local', is_flag=True)
+@click.option('-U', '--build-cache-pulled', is_flag=True)
+@click.option('-X', '--build-cache-disabled', is_flag=True)
+@click.option('--production-image', is_flag=True)
Review comment:
Let me try first. If i didn't get success, i will get your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054376653
Few comments for the "build output":
1) I think we need a bit more spacing between the lines. This is a bit to diffilcult to separate them. I think every type of message should be separated by empty line:
2) The question could be printed in yellow to drag attention.
3) Default action on timeout should be "No" and is "Quit".
4) We do not need (`y for Yes, n for N, q for Quit`) . y/n/q is pretty universal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024938554
> They are created when `--backend mssql` is used
@potiuk Here irrespective of the backend chosen we create the file in cache. Am i missing to notice any specific code flow?
https://github.com/apache/airflow/blob/1d4b709e20b07c6f0b5d1bab1935e19557df2913/scripts/ci/libraries/_initialization.sh#L45-L62
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025089492
> Do we have to use the bash commands you have used in lines 924 and 935 via subcommand in python? Or do we have to try a different way? Also, I don't understand the comment added `In GitHub Code QL, the version of docker has +azure suffix which we should remove`. Could you explain to me that part?
Yeah. We should run the `docker commands via subprocess`. We **could** potentially use docker API in python, but running docker as `docker` command is precisely what we want to check . In this check we want to check if the user has:
* good docker command line version and it is on the PATH
* the version is properly configured - when you install docker on Linux you have to perform [additional steps](https://docs.docker.com/engine/install/linux-postinstall/) to make docker available for you user - the "permission" error is the way to check it (however it would be great to see if we have the message right, it could have changed). The way to check it is to remove yourself from docker group or run docker as a diferent user who is not in `docker` group
The best way to check it is to actually run the commands :)
> In GitHub Code QL, the version of docker has +azure suffix which we should remove.
The version check simpl checks the min major/minor version and in Python it can be written in a nicer way. The point here is that usualy the version prnted by Docker is like that:
```
docker version --format '{{.Client.Version}}'
20.10.7
```
However on of our users who run ./breeze in Azure Code QL ( which is Azure's CI system) got error that the version could not be parsed, because in Azure Code QL, they MODIFIED the version of docker they have installed and the version returened was something like:
```
docker version --format '{{.Client.Version}}'
20.10.7+azure
```
:scream: :scream: :scream: :scream: :scream:
So our version check must include also this case - when parsing the version .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025796568
@potiuk I was testing with one of the versions of docker that is lower than the minimum docker version
```
docker version --format '{{.Client.Version}}'
19.03.5
```
But when validated with semver library, it says it's a invalid version name.
```
In [2]: VersionInfo.isvalid("19.03.5")
Out[2]: False
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1028948241
@potiuk Could you explain why this has to be done?
https://github.com/apache/airflow/blob/ddb5246bd1576e2ce6abf8c80c3328d7d71a75ce/breeze#L834-L838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1029192995
> @potiuk could you explain to me what this -n check does in this bash code?
Just check if they are set. If any of those are "net empty" then rather than using local sources of Airflow, we use airlfow either from:
* PYPI (if INSTALL_AIRFLOW_VERSION or USE_AIRFLOW_VERSION are set)
* GitHub (if INSTALL_AIRFLOW_REFERENCE is used.
Note that this is something that should be separated between PROD and CI image.
* USE_AIRFLOW_VERSION is for CI image
* INSTALL_AIRFLOW_REFERENCE and INSTALL_AIRFLOW_VERSION are for PROD image.
There is one difference between them : USE_AIRFLOW_VERSION will install airflow "after" you enter the image - so the "entrypoint_ci.sh" script will reinstall airflow in this case. Where INSTALL_AIRFLOW_VERSION or INSTALL_AIRFLOW_REFERENBCE are used in PROD image during the build phase - airlfow will be installed from PyPI/GiThub as part of one of the "docker/" scripts during the `docker build` command.
In our case we are still working on "build CI image" so the INSTALL* ones you can leave for later when we implement PROD image building.
> Also, I am quite confused about what variables should be in global_constants and what has to be in the dataclass.
That's simple. The global constants should contain the variables that do not change when parameters change. The dataclass should be used when you have a "calculated" value which depends on the parameters that you passed to the "breeze" command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r799265614
##########
File path: dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
##########
@@ -53,28 +67,175 @@
]
-def get_extra_docker_flags(all: bool, airflow_sources: str) -> List:
+def get_extra_docker_flags(all: bool, selected: bool, airflow_sources: str) -> List:
# get_extra_docker_flags(False, str(airflow_source))
+ # add verbosity
EXTRA_DOCKER_FLAGS = []
if all:
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}:/opt/airflow/:cached"])
- else:
+ elif selected:
for flag in NECESSARY_HOST_VOLUMES:
EXTRA_DOCKER_FLAGS.extend(["-v", airflow_sources + flag])
+ else:
+ console.print('Skip mounting host volumes to Docker')
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}/files:/files"])
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}/dist:/dist"])
EXTRA_DOCKER_FLAGS.extend(["--rm"])
EXTRA_DOCKER_FLAGS.extend(["--env-file", f"{airflow_sources}/scripts/ci/docker-compose/_docker.env"])
return EXTRA_DOCKER_FLAGS
-def check_docker_resources(
- verbose: bool, mount_all_flag: bool, airflow_sources: str, airflow_ci_image_name: str
-):
- extra_docker_flags = get_extra_docker_flags(mount_all_flag, airflow_sources)
+def check_docker_resources(verbose: bool, airflow_sources: str, airflow_ci_image_name: str):
+ extra_docker_flags = get_extra_docker_flags(
+ MOUNT_ALL_LOCAL_SOURCES, MOUNT_SELECTED_LOCAL_SOURCES, airflow_sources
+ )
cmd = []
cmd.extend(["docker", "run", "-t"])
cmd.extend(extra_docker_flags)
cmd.extend(["--entrypoint", "/bin/bash", airflow_ci_image_name])
cmd.extend(["-c", "python /opt/airflow/scripts/in_container/run_resource_check.py"])
run_command(cmd, verbose=verbose, text=True)
+
+
+def check_docker_permission(verbose) -> bool:
+ permission_denied = False
+ docker_permission_command = ["docker", "info"]
+ try:
+ _ = run_command(
+ docker_permission_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ except subprocess.CalledProcessError as ex:
+ permission_denied = True
+ if ex.stdout and 'Got permission denied while trying to connect' in ex.stdout:
+ console.print('ERROR: You have `permission denied` error when trying to communicate with docker.')
+ console.print(
+ 'Most likely you need to add your user to `docker` group: \
+ https://docs.docker.com/ engine/install/linux-postinstall/ .'
+ )
+ return permission_denied
+
+
+def compare_version(current_version: str, min_version: str) -> bool:
+ return version.parse(current_version) > version.parse(min_version)
+
+
+def check_docker_version(verbose: bool):
+ permission_denied = check_docker_permission(verbose)
+ if not permission_denied:
+ docker_version_command = ['docker', 'version', '--format', '{{.Client.Version}}']
+ docker_version = ''
+ docker_version_output = run_command(
+ docker_version_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ if docker_version_output.returncode == 0:
+ docker_version = docker_version_output.stdout.strip()
+ if docker_version == '':
+ console.print(
+ f'Your version of docker is unknown. If the scripts fail, please make sure to \
+ install docker at least: {MIN_DOCKER_VERSION} version.'
+ )
+ else:
+ good_version = compare_version(docker_version, MIN_DOCKER_VERSION)
+ if good_version:
+ console.print(f'Good version of Docker: {docker_version}.')
+ else:
+ console.print(
+ f'Your version of docker is too old:{docker_version}. Please upgrade to \
+ at least ${MIN_DOCKER_VERSION}'
+ )
+
+
+def check_docker_compose_version(verbose: bool):
+ version_pattern = re.compile(r'(\d+)\.(\d+)\.(\d+)')
+ docker_compose_version_command = ["docker-compose", "--version"]
+ docker_compose_version_output = run_command(
+ docker_compose_version_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ if docker_compose_version_output.returncode == 0:
+ docker_compose_version = docker_compose_version_output.stdout
+ version_extracted = version_pattern.search(docker_compose_version)
+ if version_extracted is not None:
+ version = '.'.join(version_extracted.groups())
+ good_version = compare_version(version, MIN_DOCKER_COMPOSE_VERSION)
+ if good_version:
+ console.print(f'Good version of docker-compose: {version}')
+ else:
+ console.print(
+ f'You have too old version of docker-compose: {version}! \
+ At least 1.29 is needed! Please upgrade!'
+ )
+ console.print(
+ 'See https://docs.docker.com/compose/install/ for instructions. \
+ Make sure docker-compose you install is first on the PATH variable of yours.'
+ )
+ else:
+ console.print(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+def prepare_docker_compose_files(verbose: bool, backend: str, integrations: Tuple[str]):
+ # fetch backend and integration from cache
+ compose_ci_file = []
+ main_ci_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/base.yml"
+ backend_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/backend-{backend}.yml"
+ backend_port_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/backend-{backend}-port.yml"
+ local_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/local.yml"
+ local_all_sources_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/local-all-sources.yml"
+ files_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/files.yml"
+ remove_sources_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/remove-sources.yml"
+ forward_credentials_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/forward-credentials.yml"
+ # mssql based check have to be added
+ if backend == 'mssql':
+ docker_filesystem_command = [
Review comment:
@potiuk I face error when executing this command via subprocess. While it works via terminal, it throws error when executed via subprocess. Can you help me to solve this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025659012
You can read more about the bash magic you can do with variable substitution (this is the exact name of what we do here) https://tldp.org/LDP/abs/html/parameter-substitution.html
Also what's part of the magic here:
* `${1//[.-]/ }` - the 'space' between `/` and `}` matter - this is how we get the parameters to be separated - basically we replace any "." or "-" with space with this construct. The '//' means that this is "global" replacement - so it will replace "all" . and - found not only the first (`${1/[.-]/ }` only replaces the first it finds).
* It is important that there is no `"` around it. The `printf "%03d%03d%03d%.0s" "${1//[.-]/ }"` would not work because the "2 20 1" would be seen as a single parameter, but if you do not add `"` the space we added in the previous step, makes each part of the number a separate parameter to `printf`. This is kinda unusual use and usually you want those quotes. This is why shellcheck (our pre-commit static check for bash code) complained and this is why I had to add ` # shellcheck disable=SC2086,SC2183 ` to stop it from complaining.
Now I think you should know why we want to get rid of Bash :). If explaining one line of bash takes 2-3 paragraph of text, it means that we should get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025659012
You can read more about the bash magic you can do with variable substitution (this is the exact name of what we do here) https://tldp.org/LDP/abs/html/parameter-substitution.html
Also what's part of the magic here:
* `${1//[.-]/ }` - the 'space' between `/` and `}` matter - this is how we get the parameters to be separated - basically we replace any `.` or `-` with space with this construct. The `//` means that this is "global" replacement - so it will replace "all" `.` and `-` found not only the first (`${1/[.-]/ }` only replaces the first it finds).
* It is important that there is no `"` around it. The `printf "%03d%03d%03d%.0s" "${1//[.-]/ }"` would not work because the "2 20 1" would be seen as a single parameter, but if you do not add `"` the space we added in the previous step, makes each part of the number a separate parameter to `printf`. This is kinda unusual use and usually you want those quotes. This is why shellcheck (our pre-commit static check for bash code) complained and this is why I had to add ` # shellcheck disable=SC2086,SC2183 ` to stop it from complaining.
Now I think you should know why we want to get rid of Bash :). If explaining one line of bash takes 2-3 paragraph of text, it means that we should get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025702462
> This way we can simply compare two versions using bash arithmetics `$(( ver1 < ver2 ))` - because those are properly comparable numbers (2.2.1 < 2.20.1) and (2.20.1-azure == 2.20.1).
>
> In Python we can I think use `semver` library to make the comparision (but we have to see if numbers with `-azure` suffix
@potiuk in semver library for the comparison major, minor, patch and prerelease part are equal.
```
In [5]: semver.compare('2.20.1' , '2.20.1-azure')
Out[5]: 1
```
I will pick only major, minor and patch to do comparison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038210557
Still things pending in this task are:
- [x] Fix file permission (need more clarity on this part)
- [x] Building ci image instead of showing instructions to do it (which is currently in the code)
- [ ] ~~Production image true specific changes. Currently only ci specific cases handled~~
- [ ] Adding trap
- [x] Adding relevant test
If there is anything missing other than this part, please let me know @potiuk . Meanwhile, you can check the code, execute it and share your feedback too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038210557
Still things pending in this task are:
- [x] Fix file permission (need more clarity on this part)
- [ ] Building ci image instead of showing instructions to do it (which is currently in the code)
- [ ] ~~Production image true specific changes. Currently only ci specific cases handled~~
- [ ] Adding trap
- [ ] Adding relevant test
If there is anything missing other than this part, please let me know @potiuk . Meanwhile, you can check the code, execute it and share your feedback too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038979622
@potiuk I could see that chmod permission is done in symbolic mode. Could this be done in octal mode?
https://github.com/apache/airflow/blob/28378d867afaac497529bd2e1d2c878edf66f460/scripts/ci/libraries/_permissions.sh#L42-L53
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1035006093
> @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
Execute the command. The flow should be like that:
1) User runs ./breeze without `--force-build`:
* if the image is "fresh" (important files have not changed) - just start breeze shell without asking any questions
* if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
y -> build (and the user continues to shell after it is finished).
n -> the user continues to shell without building
q -> quit
2) User runs ./breeze with `--force-build`:
* no matter what the "build is started" first (and the user continues to shell after it is finished).
There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
If the user is not rebased, then we print the warning "This might take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r805174798
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
@potiuk this is how I tried to implement running docker-compose command with `-e` flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r805174798
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
@potiuk this is how I tried to implement running docker-compose command with `-e` flags.
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
sure @potiuk I will first solve the issue by fixing it in os.environ and then handle the verbose case too. thanks
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
@potiuk Do you think it's better to log all the env variable used during this docker-compose command?
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
ok @potiuk i have added this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1027121665
@potiuk could you explain to me this part? I can understand that we check if the files in the list of `FILES_FOR_REBUILD_CHECK` are modified. we use md5sum to determine that. we persist the old md5sum calculated in build cache and if it varies from the recent file, we assume that file is modified and do build the local image. But why do we have to calculate them for only specific files in that list?
Even with docker buildx coming, we will still use this to determine if we have to build the image or not. Am i right ?
https://github.com/apache/airflow/blob/2f4a3d4d4008a95fc36971802c514fef68e8a5d4/scripts/ci/libraries/_md5sum.sh#L98-L110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025651448
BTW. You can run the printf examples above and see yourself :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025659012
You can read more about the bash magic you can do with variable substitution (this is the exact name of what we do here) https://tldp.org/LDP/abs/html/parameter-substitution.html
Also what's part of the magic here:
* `${1//[.-]/ }` - the 'space' between `/` and `}` matter - this is how we get the parameters to be separated - basically we replace any "." or "-" with space with this construct.
* It is important that there is no `"` around it. The `printf "%03d%03d%03d%.0s" "${1//[.-]/ }"` would not work because the "2 20 1" would be seen as a single parameter, but if you do not add `"` the space we added in the previous step, makes each part of the number a separate parameter to `printf`. This is kinda unusual use and usually you want those quotes. This is why shellcheck (our pre-commit static check for bash code) complained and this is why I had to add ` # shellcheck disable=SC2086,SC2183 ` to stop it from complaining.
Now I think you should know why we want to get rid of Bash :). If explaining one line of bash takes 2-3 paragraph of text, it means that we should get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025796568
@potiuk I was testing with one of the versions of docker that is lower than the minimum docker version
```
docker version --format '{{.Client.Version}}'
19.03.5
```
But when validated with semver library, it says it's a invalid version name.
```
In [2]: VersionInfo.isvalid("19.03.5")
Out[2]: False
```
https://docs.docker.com/engine/release-notes/19.03/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024892285
PRs for two ADRs added based on those questions :) https://github.com/apache/airflow/pull/21209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1027858573
> @potiuk could you explain to me this part? I can understand that we check if the files in the list of `FILES_FOR_REBUILD_CHECK` are modified. we use md5sum to determine that. we persist the old md5sum calculated in build cache and if it varies from the recent file, we assume that file is modified and do build the local image. But why do we have to calculate them for only specific files in that list?
Yeah. The thing is that we have a lot of files "mounted" inside the Dockerfile. All the sources of airflow are mounted in the place where they would normally be baked in when docker image is built. Mounting files when you run breeze is super quick (milliseconds). Re-building image on the other hand takes 20-30 seconds if you have almost nothing to build (just modified sources). That's why when you modified your file localy, you can enter breeze and run test in a matter of seconds. This allow for very quick iterations when you modify the code and want to test. Imagine if after every single change of your file, you'd have to wait 20-30 seconds to just be able to test it. That's not going to fly. And rebuilding the image is not really needed in this case because mounting the files effectively does what rebuilding the image does.
So instead - we only suggest to rebuild the files when we know the "important" files changed and mounting is not "enough":
* When Dockerfile.ci changes - you likely need to rebuild it because maybe somene added new tools/dependencies via apt
* When setup.py/setup.cfg changes - you need to rebuild it to reinstall `pip` dependencies - because someone likely added a new dependency (or maybe even you added a new dependency and you want to get it installed inside the image persistently)
* The ./docker scripts - similarly as Dockerfile.ci - they might do some new installation - if one of them is changed they influence
the docker image internals
* www/ui package/yarn/webpack- they likely require new "node_modules" to be installed and those are also installed as a separate step in Dockerfile
* . dockerignore - might mean that some new files have been excluded/added to "Docker context" when docker is build, so likelly docker build is needed to reflect those changes.
Unlike Python souce code - it's not enough to "mount" that files inside the docker to replaces the files in the image. You need to run an extra "action" afterwards if you want changes in those files to be reflected in the image: 'pip install', "yarn install", or just running the scripts to install mysql or others.
So this is purely optimization of iteration speed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1042764288
> finally looking at it now - sorry for some delays in responding.
>
> > @potiuk I am currently working on writing tests and there are many functions that execute the docker command via subprocess. How can we add tests for these functions?
>
> The best way is to mock the execution of the run_command function and check if the right command is executed. Give me an hours - I will revfew the code and add some example test
Sure @potiuk Thank you so much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1035006093
> @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
Execute the command. The flow should be like that:
1) User runs ./breeze without `--force-build`:
* if the image is "fresh" (important files have not changed) - just start breeze shell without asking any questions
* if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
y -> build (and the user continues to shell after it is finished).
n -> the user continues to shell without building
q -> quit
2) User runs ./breeze with `--force-build`:
* no matter what the "build is started" first (and the user continues to shell after it is finished).
There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
If the user is not rebased, then we print the warning "This will take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1035006093
> @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
Execute the command. The flow should be like that:
1) User runs ./breeze without `--force-build`:
* if the image is "fresh" - just start breeze shell
* if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
2) User runs ./breeze with `--force-build`:
* no matter what the "build is started"
There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
If the user is not rebased, then we print the warning "This will take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1042956328
@potiuk I face issues in build-image even after pulling the code. I am not sure if I did anything wrong? i tried in different terminal and still facing error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1030061482
> yes, I could understand this part, but figuring out if the variable remains constant / varies depending on 'parameters' passed to breeze command is the part where I am confused as you mentioned. Say example `USE_AIRFLOW_VERSION ` does it have to go to global constants or dataclass. Does the user can set this value / does it have to remain constant? How do I figure out that? @potiuk
It's passed by parameter (`--use-airflow-version` in this case) , so it should be in dataclass. You should not be able to change a constants with the command line parameter really :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1030061482
> yes, I could understand this part, but figuring out if the variable remains constant / varies depending on 'parameters' passed to breeze command is the part where I am confused as you mentioned. Say example `USE_AIRFLOW_VERSION ` does it have to go to global constants or dataclass. Does the user can set this value / does it have to remain constant? How do I figure out that? @potiuk
It's passed by parameter, so it should be in dataclass. You should not be able to change a constants with the command line parameter really :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1031448485
> @potiuk Could you tell me what are the supported values for `ENABLED_SYSTEMS`? I could see that it's disabled by default. But I am not sure what are the values that can be supported in this param.
>
> https://github.com/apache/airflow/blob/a49224fa7ce45e9765c0d752edc30430e0d3ce14/scripts/ci/libraries/_initialization.sh#L306-L308
You can skip that part completely. It is going to be completely changed by in-progress "AIP-47 System Tests" that we are working on now https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests - once we agree on it and implement it, this will simplify this part and the Breeze support for system tests will be removed (as it will not be needed any more)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1031448485
> @potiuk Could you tell me what are the supported values for `ENABLED_SYSTEMS`? I could see that it's disabled by default. But I am not sure what are the values that can be supported in this param.
>
> https://github.com/apache/airflow/blob/a49224fa7ce45e9765c0d752edc30430e0d3ce14/scripts/ci/libraries/_initialization.sh#L306-L308
You can skip that part completely. It is going to be completely changed by in--progress "AIP-47 System Tests" that we are working on now https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests - once we agree on it and implement it, this will simplify this part and the Breeze support for system tests will be removed (as it will not be needed any more)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039033267
I think the easiest aand most pythonic way is described as accepted answer here (bit manipulation):
https://stackoverflow.com/questions/25988471/remove-particular-permission-using-os-chmod
For files:
```
current = stat.S_IMODE(os.stat(file).st_mode)
new = current & ~stat.S_IWGRP & ~stat.SIWOTH # Removes group/other write permission
os.chmod(file, new)
```
For directories:
```
current = stat.S_IMODE(os.stat(dir).st_mode)
new = current & ~stat.S_IWGRP & ~stat.S_IWOTH # Removes group/other write permission
new = new | stat.S_IXGRP | stat.S_IXOTH # Add group/other execute permission (to be able to list directories)
os.chmod(dir, new)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r811960847
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
Review comment:
@potiuk I have one more doubt in this mock part. we have two print statement if the docker compose version is low. But in the mock i could verify only one. Do you have any idea how i can add both prints in the mock?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r813834570
##########
File path: dev/breeze/tests/test_run_utils.py
##########
@@ -0,0 +1,24 @@
+from pathlib import Path
+from unittest import mock
+import stat
+import os
+import pytest
+from airflow_breeze.utils.run_utils import change_directory_permission, change_file_permission
+
Review comment:
@potiuk Please share your reviews about this test code that i tried.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1034805233
> @potiuk Can you tell me if we need to handle the flag `--force-build-images`? Is it still relevant after the docker buildkit part?
Yep. If "force build" is specified we should completely skip checking if the "important" files were modified and launch "build part" regardless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1033478451
@potiuk I tried running build-ci-image command with Breeze2 and I get the following error. Is there any change that I missed to incorporate ?
```
(airflow-env) sathishkannan@Sathishs-MacBook-Air airflow % ./Breeze2 build-ci-image
[+] Building 0.8s (3/3) FINISHED
=> [internal] load build definition from Dockerfile.ci 0.0s
=> => transferring dockerfile: 41B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> ERROR [internal] load metadata for ghcr.io/apache/airflow/main/python:3.7-slim-buster 0.7s
------
> [internal] load metadata for ghcr.io/apache/airflow/main/python:3.7-slim-buster:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 403 Forbidden
========================= OUTPUT start ============================
None
None
========================= OUTPUT end ============================
Traceback (most recent call last):
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/bin/Breeze2", line 33, in <module>
sys.exit(load_entry_point('apache-airflow-breeze', 'console_scripts', 'Breeze2')())
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/sathishkannan/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/breeze.py", line 245, in build_ci_image
build_image(
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/ci/build_image.py", line 96, in build_image
output = run_command(cmd, verbose=verbose, text=True)
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/utils/run_utils.py", line 45, in run_command
return subprocess.run(cmd, check=check, **kwargs)
File "/Users/sathishkannan/.pyenv/versions/3.8.10/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker', 'build', '--build-arg', 'PYTHON_BASE_IMAGE=ghcr.io/apache/airflow/main/python:3.7-slim-buster', '--build-arg', 'AIRFLOW_VERSION=2.3.0.dev0', '--build-arg', 'AIRFLOW_BRANCH=main', '--build-arg', 'AIRFLOW_EXTRAS=devel_ci', '--build-arg', 'AIRFLOW_PRE_CACHED_PIP_PACKAGES=true', '--build-arg', 'ADDITIONAL_AIRFLOW_EXTRAS=', '--build-arg', 'ADDITIONAL_PYTHON_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_COMMAND=', '--build-arg', 'ADDITIONAL_DEV_APT_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_ENV=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_COMMAND=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_DEPS=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_ENV=', '--build-arg', 'UPGRADE_TO_NEWER_DEPENDENCIES=false', '--build-arg', 'CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow', '--build-arg', 'AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main', '--build-arg', 'AIRFLOW_CONSTRAINTS=constraints-source-providers', '--build-arg', 'AIRFLOW_IMAGE_REPOSITORY=https://github.co
m/apache/airflow', '--build-arg', 'AIRFLOW_IMAGE_DATE_CREATED=2022-02-09T13:50:13Z', '--build-arg', 'BUILD_ID=0', '--build-arg', 'COMMIT_SHA=ebf760871470446fb5e0cd58edf8fec32547e65e', '--cache-from', 'ghcr.io/apache/airflow/main/ci/python3.7:latest', '-t', 'ghcr.io/apache/airflow/main/ci/python3.7:latest', '--target', 'main', '.', '-f', '/Users/sathishkannan/code/airflow/Dockerfile.ci']' returned non-zero exit status 1.
(airflow-env) sathishkannan@Sathishs-MacBook-Air airflow %
```
Edit: @potiuk I noticed its change of buildkit. I used dry_run_docker to figure out the modified docker command to execute the build image part. I will fix it and update it. Raised a relevant issue related to this. https://github.com/apache/airflow/issues/21454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1034805796
And just `--force-build` as flag name is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024884489
> @potiuk Could you share your views about getting host user id and group id? Do we have to find its equivalent in windows to make this work? Also, can you explain to me more about why we need to find the host user id and group id and how it's used? There are a few comments about it in the code, but I couldn't fully understand it.
This is only really needed on Linux. On Windows/MacOS they are not needed and can be empty (I believe - to be checked).
The reason is that on Linux files that are mounted from the host to the container are mounted using native filesystem. This basically means that any file created inside the container will keep the userid /group id that are used in container also in the host.
For example if we have user id 50001 and group id 501 in the container, any file we create in container will keep the same user is and group id in the host. But those user/group ids might not exist in the Host. - if we create a user 50001 in the container, the id will remain like that on the host, when we exit from the container. This is very problematic on linux because when we map "logs" directory and some logs (and directories) are created there, they might be owned by a non-existing user after we exit. And we want to be able to see the logs outside of the container because that's where we usually have IDE and that's where we keep reading those and analyse them.
Then, the problem is that if you want to delete such folders and files, you need to use `sudo` in the host, because your regular user has no access to it. This is big problem especially if files are created inside your source directory (which is also mounted to the container) - for example it will prevent you from switching branches easily because git will not be able to remove some files and it will refuse to switch branches.
There is also "reverse" problem - if you create files in a host with no "all" permissions, and you mount them inside the container, and container runs as "different" user, the user in container cannot access to those file (unless you run as root inside the container - root inside the container is equivalent to root in host and can access and update all files).
This can be mitigated by "user remapping" - https://docs.docker.com/engine/security/userns-remap/ - but this can only be configure at the "docker daemon" level, and this is something we should not require an average user should do, also the problem with user remapping is that it is "global" setting. It will remap your user for all containers and in many cases this is not what you really want.
So In order to avoid that we have a few things:
a) We use `root` user in container - all the files are created and run as root user. This is not recommended for production but it is great for CI - because you can freely create and read any mounted files (no matter what user), you can also run pip/apt etc. without sudo and it is generally much more convenient for many development tasks. The side effect of that is that all files created in the container have root user/group set.
b) we pass HOST_USER_ID and HOST_GROUP_ID to the container, so that we know who is the user on the host. Depending on lthe linux distro and even depending on your configuration (how many users you have created and in which sequence) - the UID can be different.
c) when the user enters the container, we set a "trap"L `add_trap "in_container_fix_ownership" EXIT HUP INT TERM` - this trap runs "fix_ownership" script that looks for all created files in the directories where we expect we will create files:
```
"/files"
"/root/.aws"
"/root/.azure"
"/root/.config/gcloud"
"/root/.docker"
"/opt/airflow/logs"
"/opt/airflow/docs"
"/opt/airflow/dags"
"${AIRFLOW_SOURCES}"
```
Whenever we exit, or terminate the container, this script is executed and it finds all files owned by "root" in those directories and changes their ownership to be HOST_USER/HOST_GROUP. This way when you exit the containers on linux, the files are owned by the host user, and can be easily deleted - either manually or when you change branches.
I hope it is clearer now. I will create an ADR out of that comment :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1026046327
@potiuk Could we check this one for version comparison? https://packaging.pypa.io/en/latest/version.html#packaging.version.parse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025647705
> @potiuk Can you explain to me what's happening in this code part? Couldn't decipher what this regex part is doing?
Bash and printf magic combined :) (and one reason why we want to get rid of it because not many people can figure it out.
1) The `${1//[.-]/ }` splits the parameter with "." and "-" separators
* "2.20.1" -> "2" "20" "1"
* "2.2.1" -> "2" "2" "1"
* "2.20.1-azure" -> "2" "20" "1" "azure"
2) `Printf "%03d%03d%03d%.0s"` formats the number to get them into comparable number, no matter what length the "numbers" are (below 4) and removes the "string" (the `%.0s` produces empty string always - no matter what was the argument).
```
printf "%03d%03d%03d%.0s" "2" "20" "1" -> 002020001
printf "%03d%03d%03d%.0s" "2" "2" "1" -> 002002001
printf "%03d%03d%03d%.0s" "2" "20" "1" "azure" -> 002020001
```
This way we can simply compare two versions using bash arithmetics `$(( ver1 < ver2 ))` - because those are properly comparable numbers (2.2.1 < 2.20.1) and (2.20.1-azure == 2.20.1).
In Python we can I think use `semver` library to make the comparision (but we have to see if numbers with `-azure` suffix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025089492
> Do we have to use the bash commands you have used in lines 924 and 935 via subcommand in python? Or do we have to try a different way? Also, I don't understand the comment added `In GitHub Code QL, the version of docker has +azure suffix which we should remove`. Could you explain to me that part?
Yeah. We should run the `docker commands` via subprocess. We **could** potentially use docker API in python, but running docker as `docker` command is precisely what we want to check . In this check we want to check if the user has:
* good docker command line version and it is on the PATH
* the version is properly configured - when you install docker on Linux you have to perform [additional steps](https://docs.docker.com/engine/install/linux-postinstall/) to make docker available for you user - the "permission" error is the way to check it (however it would be great to see if we have the message right, it could have changed). The way to check it is to remove yourself from docker group or run docker as a diferent user who is not in `docker` group
The best way to check it is to actually run the commands :)
> In GitHub Code QL, the version of docker has +azure suffix which we should remove.
The version check simpl checks the min major/minor version and in Python it can be written in a nicer way. The point here is that usualy the version prnted by Docker is like that:
```
docker version --format '{{.Client.Version}}'
20.10.7
```
However on of our users who run ./breeze in Azure Code QL ( which is Azure's CI system) got error that the version could not be parsed, because in Azure Code QL, they MODIFIED the version of docker they have installed and the version returened was something like:
```
docker version --format '{{.Client.Version}}'
20.10.7+azure
```
:scream: :scream: :scream: :scream: :scream:
So our version check must include also this case - when parsing the version .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025712312
> Here the `+azure` would indicate the build part. So I think we don't need to remove it explicitely like how it's done using sed in bash. We can simply pass that to semver and it will work fine.
Sounds exactly what we need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1037260947
> @potiuk I am trying to run docker-compose command via subprocess and I face error when I pass env variables as -e. It works properly when set as os.environ variable and executing the docker-compose command. But it didn't work that way when I tried via `-e` flag. I will check if I could fix it.
How do you pass them ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r805338267
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
ok @potiuk i have added this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038132045
@potiuk Could you explain more about this part? I could understand it fix the permission issue, but I don't understand what problems it is causing and why it has to be fixed this way. Also do we need to implement the same by running the command in subprocess module python?
https://github.com/apache/airflow/blob/cca2f942ea9363303a041f00e0f180924cb840a9/scripts/ci/libraries/_permissions.sh#L19-L53
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038979622
@potiuk I could see that chmod permission is done in symbolic mode. Could this be done in octal mode? I have this doubt because when we do in symbolic mode we can toggle one of the group/user/other permission alone without touching others, but when we do in octal mode, it changes the entire permission.
https://github.com/apache/airflow/blob/28378d867afaac497529bd2e1d2c878edf66f460/scripts/ci/libraries/_permissions.sh#L42-L53
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038196785
Note! We cannot "blank" remove all the files in airflow directory, becuase
it can take a very long time when the users develops Airlfow. When you develop
and run airflow locally, there are a lot of files created and generated -
log files, build artifacts, but also node modules (a loooot of files) and
generated static files. Those files are not needed/used in the Docker build
(we make sure of that by removing all files by default and only selectively
adding files that we need). On a number of systems (especialy those where
the filesystem is mounted to docker container via user filesystem - MacOS
or Windows) just changing permission of all those files can change tens of
seconds - so instead we opted to change only permissions of those files
that are added to Git.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1027860655
> backend_port_docker_compose_file
Hmm. I am not entirely sure why this one. We can remove it for now and in case we find the reason why we need to add them back - we will (and document it this time :) )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1041254478
> > @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
>
> Execute the command. The flow should be like that:
>
> 1. User runs ./breeze without `--force-build`:
>
> * if the image is "fresh" (important files have not changed) - just start breeze shell without asking any questions
> * if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
> y -> build (and the user continues to shell after it is finished).
> n -> the user continues to shell without building
> q -> quit
>
> 2. User runs ./breeze with `--force-build`:
>
> * no matter what the "build is started" first (and the user continues to shell after it is finished).
>
> There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
>
> If the user is not rebased, then we print the warning "This might take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
@potiuk I have added code to support this part. But i didn't add the timeout part yet. Could you point me to the code that checks if the user is rebased to latest "main"? That part is not added too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r812722254
##########
File path: dev/breeze/src/airflow_breeze/utils/run_utils.py
##########
@@ -73,3 +97,110 @@ def check_package_installed(package_name: str) -> bool:
Please install using https://pre-commit.com/#install to continue[/]\n"
)
return is_installed
+
+
+def get_filesystem_type(filepath):
+ root_type = "unknown"
+ for part in psutil.disk_partitions():
+ if part.mountpoint == '/':
+ root_type = part.fstype
+ continue
+ if filepath.startswith(part.mountpoint):
+ return part.fstype
+
+ return root_type
+
+
+def calculate_md5_checksum_for_files(md5sum_cache_dir: Path):
+ not_modified_files = []
+ modified_files = []
+ for calculate_md5_file in FILES_FOR_REBUILD_CHECK:
+ file_to_get_md5 = Path(AIRFLOW_SOURCE, calculate_md5_file)
+ md5_checksum = generate_md5(file_to_get_md5)
+ sub_dir_name = file_to_get_md5.parts[-2]
+ actual_file_name = file_to_get_md5.parts[-1]
+ cache_file_name = Path(md5sum_cache_dir, sub_dir_name + '-' + actual_file_name + '.md5sum')
+ file_content = md5_checksum + ' ' + str(file_to_get_md5) + '\n'
+ is_modified = update_md5checksum_in_cache(file_content, cache_file_name)
+ if is_modified:
+ modified_files.append(calculate_md5_file)
+ else:
+ not_modified_files.append(calculate_md5_file)
+ return modified_files, not_modified_files
+
+
+def md5sum_check_if_build_is_needed(md5sum_cache_dir: Path, the_image_type: str) -> bool:
+ build_needed = False
+ modified_files, not_modified_files = calculate_md5_checksum_for_files(md5sum_cache_dir)
+ if len(modified_files) > 0:
+ console.print('The following files are modified: ', modified_files)
+ console.print(f'Likely {the_image_type} image needs rebuild')
+ build_needed = True
+ else:
+ console.print(
+ f'Docker image build is not needed for {the_image_type} build as no important files are changed!'
+ )
+ return build_needed
+
+
+def instruct_build_image(the_image_type: str, python_version: str):
+ console.print(f'The {the_image_type} image for python version {python_version} may be outdated')
+ console.print('Please run this command at earliest convenience:')
+ if the_image_type == 'CI':
+ console.print(f'./Breeze2 build-ci-image --python {python_version}')
+ else:
+ console.print(f'./Breeze2 build-prod-image --python {python_version}')
+ console.print("If you run it via pre-commit as individual hook, you can run 'pre-commit run build'.")
+
+
+def instruct_for_setup():
+ CMDNAME = 'Breeze2'
+ console.print(f"You can setup autocomplete by running {CMDNAME} setup-autocomplete'")
+ console.print(" You can toggle ascii/cheatsheet by running:")
+ console.print(f" * {CMDNAME} toggle-suppress-cheatsheet")
+ console.print(f" * {CMDNAME} toggle-suppress-asciiart")
+
+
+@contextlib.contextmanager
+def working_directory(source_path: Path):
+ # Equivalent of pushd and popd in bash script.
+ # https://stackoverflow.com/a/42441759/3101838
+ prev_cwd = Path.cwd()
+ os.chdir(source_path)
+ try:
+ yield
+ finally:
+ os.chdir(prev_cwd)
+
+
+def change_file_permission(file_to_fix: Path):
Review comment:
@potiuk Could you tell me if i can use mock to test the `change_file_permission` part too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #21145:
URL: https://github.com/apache/airflow/pull/21145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1029581924
> That's simple. The global constants should contain the variables that do not change when parameters change. The dataclass should be used when you have a "calculated" value which depends on the parameters that you passed to the "breeze" command.
yes, I could understand this part, but figuring out if the variable remains constant / varies depending on 'parameters' passed to breeze command is the part where I am confused. Say example `USE_AIRFLOW_VERSION ` does it have to go to global constants or dataclass. Does user can set this value / it has to remain constant? How do i figure out that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1041254478
> > @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
>
> Execute the command. The flow should be like that:
>
> 1. User runs ./breeze without `--force-build`:
>
> * if the image is "fresh" (important files have not changed) - just start breeze shell without asking any questions
> * if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
> y -> build (and the user continues to shell after it is finished).
> n -> the user continues to shell without building
> q -> quit
>
> 2. User runs ./breeze with `--force-build`:
>
> * no matter what the "build is started" first (and the user continues to shell after it is finished).
>
> There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
>
> If the user is not rebased, then we print the warning "This might take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
@potiuk I have added code to support this part. But I didn't add the timeout part yet. After the timeout, how do we need to proceed? Do we need to take that as yes/no from the user? Could you point me to the code that checks if the user is rebased to the latest "main"? That part is not added too.
Also build image command is added but it's not yet working properly due to issue https://github.com/apache/airflow/issues/21454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808904082
##########
File path: dev/breeze/src/airflow_breeze/shell/shell_builder.py
##########
@@ -206,7 +205,7 @@ def compose_files(self):
)
# mssql based check have to be added
if self.backend == 'mssql':
- docker_filesystem = get_filesystem_type()
+ docker_filesystem = get_filesystem_type('.')
Review comment:
There was a filesystem missing here. I think "." will be fine, but somethign to check on CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r810231042
##########
File path: dev/breeze/src/airflow_breeze/ci/build_params.py
##########
@@ -106,8 +116,8 @@ def commit_sha(self):
def docker_cache_ci_directive(self) -> List:
docker_cache_ci_directive = []
if self.docker_cache == "pulled":
- docker_cache_ci_directive.append("--cache-from")
- docker_cache_ci_directive.append(self.airflow_ci_image_name)
+ docker_cache_ci_directive.append(f"--cache-from={self.airflow_ci_image_name_with_cache}")
Review comment:
@potiuk Changes in cache-from flag in docker command construction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r810230692
##########
File path: dev/breeze/src/airflow_breeze/ci/build_image.py
##########
@@ -83,11 +83,17 @@ def construct_docker_command(ci_image: BuildParams) -> List[str]:
def build_image(verbose, **kwargs):
ci_image_params = BuildParams(**filter_out_none(**kwargs))
+ ci_image_cache_dir = Path(BUILD_CACHE_DIR, ci_image_params.airflow_branch)
+ ci_image_cache_dir.mkdir(parents=True, exist_ok=True)
+ touch_cache_file(
+ f"built_{ci_image_params.python_version}",
+ root_dir=ci_image_cache_dir,
+ )
is_cached, value = check_cache_and_write_if_not_cached(
"PYTHON_MAJOR_MINOR_VERSION", ci_image_params.python_version
)
if is_cached:
ci_image_params.python_version = value
cmd = construct_docker_command(ci_image_params)
- output = run_command(cmd, verbose=verbose, text=True)
+ output = run_command(cmd, verbose=verbose, cwd=AIRFLOW_SOURCE, text=True)
Review comment:
@potiuk Changes done for CWD in subprocess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1035006093
> @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
Execute the command. The flow should be like that:
1) User runs ./breeze without `--force-build`:
* if the image is "fresh" (important files have not changed) - just start breeze shell without asking any questions
* if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
y -> build
n -> continue without building
q -> quit
2) User runs ./breeze with `--force-build`:
* no matter what the "build is started"
There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
If the user is not rebased, then we print the warning "This will take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1032492887
@potiuk How do i find out the flags supported for the breeze enter the shell command? I could find this in help, but it says most flags are used. Do we any list of supported flags or can i add all the flags that are used in this command?
```
Detailed usage for command: shell
breeze shell [FLAGS] [-- <EXTRA_ARGS>]
This is default subcommand if no subcommand is used.
Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
workers, interact with the database, run DAGs etc. It is the default command if no command
is selected. The shell is executed in the container and in case integrations are chosen,
the integrations will be started as separated docker containers - under the docker-compose
supervision. Local sources are by default mounted to within the container so you can edit
them locally and run tests immediately in the container. Several folders ('files', 'dist')
are also mounted so that you can exchange files between the host and container.
The 'files/airflow-breeze-config/variables.env' file can contain additional variables
and setup. This file is automatically sourced when you enter the container. Database
and webserver ports are forwarded to appropriate database/webserver so that you can
connect to it from your host environment.
You can also pass <EXTRA_ARGS> after -- they will be passed as bash parameters, this is
especially useful to pass bash options, for example -c to execute command:
'breeze shell -- -c "ls -la"'
'breeze -- -c "ls -la"'
For GitHub repository, the --github-repository flag can be used to specify the repository
to pull and push images. You can also use --github-image-id <COMMIT_SHA> in case
you want to pull the image with specific COMMIT_SHA tag.
'breeze shell \
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
'breeze \
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
Most flags are applicable to the shell command as it will run build when needed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r811626590
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
+ mock_console.print.assert_called_with('See https://docs.docker.com/compose/install/ for instructions. Make sure docker-compose you install is first on the PATH variable of yours.')
+
+
Review comment:
oh, I see. I will check about integrating unit tests with IDE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r811296904
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
+ mock_console.print.assert_called_with('See https://docs.docker.com/compose/install/ for instructions. Make sure docker-compose you install is first on the PATH variable of yours.')
+
+
Review comment:
So the tests actually detected a potential error case - if 1.29 was passed - it neither printed "good" nor "unknown" (which I would expect actually in this case).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r811295202
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
+ mock_console.print.assert_called_with('See https://docs.docker.com/compose/install/ for instructions. Make sure docker-compose you install is first on the PATH variable of yours.')
+
+
Review comment:
So `version_extracted`was None and it actualy never printed anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r812797300
##########
File path: dev/breeze/tests/test_cache.py
##########
@@ -62,6 +67,17 @@ def test_read_from_cache_file(param):
if param_value is None:
assert None is param_value
else:
- allowed, param_list = check_if_values_allowed(param + 's', param_value)
+ allowed, param_list = check_if_values_allowed(param, param_value)
if allowed:
assert param_value in param_list
+
+
Review comment:
@potiuk Could you share your views on the above question too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r816105059
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import sys
+from pathlib import Path
+from typing import Dict
+
+import click
+from inputimeout import TimeoutOccurred, inputimeout
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.ci.build_image import build_image
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ fix_group_permissions,
+ get_latest_sha,
+ instruct_build_image,
+ instruct_for_setup,
+ is_repo_rebased,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_env_variables_docker_compose_command(shell_params: ShellBuilder) -> Dict[str, str]:
+ env_variables: Dict[str, str] = {}
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ env_variables[param_name] = str(getattr(shell_params, param_value))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ env_variables[constant_param_name] = str(constant_param_value)
+ return env_variables
+
+
+def build_image_if_needed_steps(verbose: bool, shell_params: ShellBuilder):
+ build_needed = md5sum_check_if_build_is_needed(shell_params.md5sum_cache_dir, shell_params.the_image_type)
+ build_needed = True
+ if build_needed:
+ try:
+ user_status = inputimeout(
+ prompt='\nDo you want to build image?Press y/n/q in 5 seconds\n',
Review comment:
👍 Cool I added color codes explicitely. i will remove it then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r816104426
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import sys
+from pathlib import Path
+from typing import Dict
+
+import click
+from inputimeout import TimeoutOccurred, inputimeout
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.ci.build_image import build_image
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ fix_group_permissions,
+ get_latest_sha,
+ instruct_build_image,
+ instruct_for_setup,
+ is_repo_rebased,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_env_variables_docker_compose_command(shell_params: ShellBuilder) -> Dict[str, str]:
+ env_variables: Dict[str, str] = {}
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ env_variables[param_name] = str(getattr(shell_params, param_value))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ env_variables[constant_param_name] = str(constant_param_value)
+ return env_variables
+
+
+def build_image_if_needed_steps(verbose: bool, shell_params: ShellBuilder):
+ build_needed = md5sum_check_if_build_is_needed(shell_params.md5sum_cache_dir, shell_params.the_image_type)
+ build_needed = True
+ if build_needed:
+ try:
+ user_status = inputimeout(
+ prompt='\nDo you want to build image?Press y/n/q in 5 seconds\n',
Review comment:
Ah yeah. No it should be fine without colors then, :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r816105919
##########
File path: dev/breeze/tests/test_cache.py
##########
@@ -62,6 +67,17 @@ def test_read_from_cache_file(param):
if param_value is None:
assert None is param_value
else:
- allowed, param_list = check_if_values_allowed(param + 's', param_value)
+ allowed, param_list = check_if_values_allowed(param, param_value)
if allowed:
assert param_value in param_list
+
+
Review comment:
Ah missed that one. let me see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1052008832
> @potiuk I could see that fix ownership is invoked only as part of PRODUCTION_IMAGE. Can you explain me why it has to be done that way?
>
> https://github.com/apache/airflow/blob/608b8c4879c188881e057e6318a0a15f54c55c7b/breeze#L3475-L3484
Becauss in CI image we set a "trap" in "entrypoint_ci" to fix the ownership (it happens when you exit or send termination signal). So this happens automatically for CI image:
https://github.com/apache/airflow/blob/main/dev/breeze/doc/adr/0006-using-root-user-and-fixing-ownership-for-ci-container.md#decision
In production image we do not do it automatically because it might not be the intention when production image is used in "production" so we do it only if we enter prod image via breeze (which is clear is a "development" setting).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1022935412
@potiuk I want to know which is the correct set of supported Postgres versions.
https://github.com/apache/airflow/blob/6405d8f804e7cbd1748aa7eed65f2bbf0fcf022e/breeze-complete#L36
https://github.com/apache/airflow/blob/6405d8f804e7cbd1748aa7eed65f2bbf0fcf022e/scripts/ci/libraries/_initialization.sh#L121
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1023202231
> @potiuk i can see several env variables in scripts/ci/libraries/_initialization.sh.
We definitely should not try to set all of those. They are needed by different parts of "old breeze". The variables that we should pass are exactly those which are "generated" in the "dc_ci" script - just take a look what you have in `./build/dc_ci` and those are the only environment variables that are really needed to run the CI. Another place (we have a few of those) that you can reconcile are in `scripts/ci/docker-compose/_docker.env` and the same variables are repeated in `scripts/ci/docker-compose/base.yaml` - maybe there are some slight differences we will have to reconcile.
BTW. The reason why we have it in many places is that there are some difference in how "docker-compose 1" and "docker-compose 2" work. This will likely go away soon (Docker-compose 2 has been introduced recently and I think we will soon be able to deprecate docker-compose 1) at which time we will be able to remove one of those lists.
I think (can't remember) I have a pre-commit that keeps those in-check - but maybe I have not done that, so better to check and reconcile it now.
Regardless-having yet another list of the variables in python in "global_constants" that will be the "source of truth" would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024847158
@potiuk Could you share your views about getting host user id and group id? Do we have to find its equivalent in windows to make this work? Also, can you explain to me more about why we need to find the host user id and group id and how it's used?
https://github.com/apache/airflow/blob/1d4b709e20b07c6f0b5d1bab1935e19557df2913/scripts/ci/libraries/_initialization.sh#L318-L345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024847158
@potiuk Could you share your views about getting host user id and group id? Do we have to find its equivalent in windows to make this work? Also, can you explain to me more about why we need to find the host user id and group id and how it's used? There are a few comments about it in the code, but I couldn't fully understand it.
https://github.com/apache/airflow/blob/1d4b709e20b07c6f0b5d1bab1935e19557df2913/scripts/ci/libraries/_initialization.sh#L318-L345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025089894
> So our version check must include also this case - when parsing the version .
BTW. This is a great candidate for a unit test with `parameterized` approach - where you write a function that parses the version, including the `+azure` case and checking if the parsing works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r799537807
##########
File path: dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
##########
@@ -53,28 +67,175 @@
]
-def get_extra_docker_flags(all: bool, airflow_sources: str) -> List:
+def get_extra_docker_flags(all: bool, selected: bool, airflow_sources: str) -> List:
# get_extra_docker_flags(False, str(airflow_source))
+ # add verbosity
EXTRA_DOCKER_FLAGS = []
if all:
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}:/opt/airflow/:cached"])
- else:
+ elif selected:
for flag in NECESSARY_HOST_VOLUMES:
EXTRA_DOCKER_FLAGS.extend(["-v", airflow_sources + flag])
+ else:
+ console.print('Skip mounting host volumes to Docker')
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}/files:/files"])
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}/dist:/dist"])
EXTRA_DOCKER_FLAGS.extend(["--rm"])
EXTRA_DOCKER_FLAGS.extend(["--env-file", f"{airflow_sources}/scripts/ci/docker-compose/_docker.env"])
return EXTRA_DOCKER_FLAGS
-def check_docker_resources(
- verbose: bool, mount_all_flag: bool, airflow_sources: str, airflow_ci_image_name: str
-):
- extra_docker_flags = get_extra_docker_flags(mount_all_flag, airflow_sources)
+def check_docker_resources(verbose: bool, airflow_sources: str, airflow_ci_image_name: str):
+ extra_docker_flags = get_extra_docker_flags(
+ MOUNT_ALL_LOCAL_SOURCES, MOUNT_SELECTED_LOCAL_SOURCES, airflow_sources
+ )
cmd = []
cmd.extend(["docker", "run", "-t"])
cmd.extend(extra_docker_flags)
cmd.extend(["--entrypoint", "/bin/bash", airflow_ci_image_name])
cmd.extend(["-c", "python /opt/airflow/scripts/in_container/run_resource_check.py"])
run_command(cmd, verbose=verbose, text=True)
+
+
+def check_docker_permission(verbose) -> bool:
+ permission_denied = False
+ docker_permission_command = ["docker", "info"]
+ try:
+ _ = run_command(
+ docker_permission_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ except subprocess.CalledProcessError as ex:
+ permission_denied = True
+ if ex.stdout and 'Got permission denied while trying to connect' in ex.stdout:
+ console.print('ERROR: You have `permission denied` error when trying to communicate with docker.')
+ console.print(
+ 'Most likely you need to add your user to `docker` group: \
+ https://docs.docker.com/ engine/install/linux-postinstall/ .'
+ )
+ return permission_denied
+
+
+def compare_version(current_version: str, min_version: str) -> bool:
+ return version.parse(current_version) > version.parse(min_version)
+
+
+def check_docker_version(verbose: bool):
+ permission_denied = check_docker_permission(verbose)
+ if not permission_denied:
+ docker_version_command = ['docker', 'version', '--format', '{{.Client.Version}}']
+ docker_version = ''
+ docker_version_output = run_command(
+ docker_version_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ if docker_version_output.returncode == 0:
+ docker_version = docker_version_output.stdout.strip()
+ if docker_version == '':
+ console.print(
+ f'Your version of docker is unknown. If the scripts fail, please make sure to \
+ install docker at least: {MIN_DOCKER_VERSION} version.'
+ )
+ else:
+ good_version = compare_version(docker_version, MIN_DOCKER_VERSION)
+ if good_version:
+ console.print(f'Good version of Docker: {docker_version}.')
+ else:
+ console.print(
+ f'Your version of docker is too old:{docker_version}. Please upgrade to \
+ at least ${MIN_DOCKER_VERSION}'
+ )
+
+
+def check_docker_compose_version(verbose: bool):
+ version_pattern = re.compile(r'(\d+)\.(\d+)\.(\d+)')
+ docker_compose_version_command = ["docker-compose", "--version"]
+ docker_compose_version_output = run_command(
+ docker_compose_version_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ if docker_compose_version_output.returncode == 0:
+ docker_compose_version = docker_compose_version_output.stdout
+ version_extracted = version_pattern.search(docker_compose_version)
+ if version_extracted is not None:
+ version = '.'.join(version_extracted.groups())
+ good_version = compare_version(version, MIN_DOCKER_COMPOSE_VERSION)
+ if good_version:
+ console.print(f'Good version of docker-compose: {version}')
+ else:
+ console.print(
+ f'You have too old version of docker-compose: {version}! \
+ At least 1.29 is needed! Please upgrade!'
+ )
+ console.print(
+ 'See https://docs.docker.com/compose/install/ for instructions. \
+ Make sure docker-compose you install is first on the PATH variable of yours.'
+ )
+ else:
+ console.print(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+def prepare_docker_compose_files(verbose: bool, backend: str, integrations: Tuple[str]):
+ # fetch backend and integration from cache
+ compose_ci_file = []
+ main_ci_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/base.yml"
+ backend_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/backend-{backend}.yml"
+ backend_port_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/backend-{backend}-port.yml"
+ local_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/local.yml"
+ local_all_sources_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/local-all-sources.yml"
+ files_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/files.yml"
+ remove_sources_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/remove-sources.yml"
+ forward_credentials_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/forward-credentials.yml"
+ # mssql based check have to be added
+ if backend == 'mssql':
+ docker_filesystem_command = [
Review comment:
I need to see it - ping me on slack or better schedule 1-1 session, I think this is about the right time to have a session with entering the shell as I can help to resolve it quicker by chatting and sharing screens :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r799537807
##########
File path: dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
##########
@@ -53,28 +67,175 @@
]
-def get_extra_docker_flags(all: bool, airflow_sources: str) -> List:
+def get_extra_docker_flags(all: bool, selected: bool, airflow_sources: str) -> List:
# get_extra_docker_flags(False, str(airflow_source))
+ # add verbosity
EXTRA_DOCKER_FLAGS = []
if all:
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}:/opt/airflow/:cached"])
- else:
+ elif selected:
for flag in NECESSARY_HOST_VOLUMES:
EXTRA_DOCKER_FLAGS.extend(["-v", airflow_sources + flag])
+ else:
+ console.print('Skip mounting host volumes to Docker')
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}/files:/files"])
EXTRA_DOCKER_FLAGS.extend(["-v", f"{airflow_sources}/dist:/dist"])
EXTRA_DOCKER_FLAGS.extend(["--rm"])
EXTRA_DOCKER_FLAGS.extend(["--env-file", f"{airflow_sources}/scripts/ci/docker-compose/_docker.env"])
return EXTRA_DOCKER_FLAGS
-def check_docker_resources(
- verbose: bool, mount_all_flag: bool, airflow_sources: str, airflow_ci_image_name: str
-):
- extra_docker_flags = get_extra_docker_flags(mount_all_flag, airflow_sources)
+def check_docker_resources(verbose: bool, airflow_sources: str, airflow_ci_image_name: str):
+ extra_docker_flags = get_extra_docker_flags(
+ MOUNT_ALL_LOCAL_SOURCES, MOUNT_SELECTED_LOCAL_SOURCES, airflow_sources
+ )
cmd = []
cmd.extend(["docker", "run", "-t"])
cmd.extend(extra_docker_flags)
cmd.extend(["--entrypoint", "/bin/bash", airflow_ci_image_name])
cmd.extend(["-c", "python /opt/airflow/scripts/in_container/run_resource_check.py"])
run_command(cmd, verbose=verbose, text=True)
+
+
+def check_docker_permission(verbose) -> bool:
+ permission_denied = False
+ docker_permission_command = ["docker", "info"]
+ try:
+ _ = run_command(
+ docker_permission_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ except subprocess.CalledProcessError as ex:
+ permission_denied = True
+ if ex.stdout and 'Got permission denied while trying to connect' in ex.stdout:
+ console.print('ERROR: You have `permission denied` error when trying to communicate with docker.')
+ console.print(
+ 'Most likely you need to add your user to `docker` group: \
+ https://docs.docker.com/ engine/install/linux-postinstall/ .'
+ )
+ return permission_denied
+
+
+def compare_version(current_version: str, min_version: str) -> bool:
+ return version.parse(current_version) > version.parse(min_version)
+
+
+def check_docker_version(verbose: bool):
+ permission_denied = check_docker_permission(verbose)
+ if not permission_denied:
+ docker_version_command = ['docker', 'version', '--format', '{{.Client.Version}}']
+ docker_version = ''
+ docker_version_output = run_command(
+ docker_version_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ if docker_version_output.returncode == 0:
+ docker_version = docker_version_output.stdout.strip()
+ if docker_version == '':
+ console.print(
+ f'Your version of docker is unknown. If the scripts fail, please make sure to \
+ install docker at least: {MIN_DOCKER_VERSION} version.'
+ )
+ else:
+ good_version = compare_version(docker_version, MIN_DOCKER_VERSION)
+ if good_version:
+ console.print(f'Good version of Docker: {docker_version}.')
+ else:
+ console.print(
+ f'Your version of docker is too old:{docker_version}. Please upgrade to \
+ at least ${MIN_DOCKER_VERSION}'
+ )
+
+
+def check_docker_compose_version(verbose: bool):
+ version_pattern = re.compile(r'(\d+)\.(\d+)\.(\d+)')
+ docker_compose_version_command = ["docker-compose", "--version"]
+ docker_compose_version_output = run_command(
+ docker_compose_version_command,
+ verbose=verbose,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ if docker_compose_version_output.returncode == 0:
+ docker_compose_version = docker_compose_version_output.stdout
+ version_extracted = version_pattern.search(docker_compose_version)
+ if version_extracted is not None:
+ version = '.'.join(version_extracted.groups())
+ good_version = compare_version(version, MIN_DOCKER_COMPOSE_VERSION)
+ if good_version:
+ console.print(f'Good version of docker-compose: {version}')
+ else:
+ console.print(
+ f'You have too old version of docker-compose: {version}! \
+ At least 1.29 is needed! Please upgrade!'
+ )
+ console.print(
+ 'See https://docs.docker.com/compose/install/ for instructions. \
+ Make sure docker-compose you install is first on the PATH variable of yours.'
+ )
+ else:
+ console.print(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+def prepare_docker_compose_files(verbose: bool, backend: str, integrations: Tuple[str]):
+ # fetch backend and integration from cache
+ compose_ci_file = []
+ main_ci_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/base.yml"
+ backend_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/backend-{backend}.yml"
+ backend_port_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/backend-{backend}-port.yml"
+ local_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/local.yml"
+ local_all_sources_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/local-all-sources.yml"
+ files_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/files.yml"
+ remove_sources_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/remove-sources.yml"
+ forward_credentials_docker_compose_file = f"{str(SCRIPTS_CI_DIR)}/docker-compose/forward-credentials.yml"
+ # mssql based check have to be added
+ if backend == 'mssql':
+ docker_filesystem_command = [
Review comment:
I need to see it - ping me on slack or better schedule 1-1 session, I think this is about the right time to have a session with entering the shell as I can help to resolve it quicker by chatting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1031448485
> @potiuk Could you tell me what are the supported values for `ENABLED_SYSTEMS`? I could see that it's disabled by default. But I am not sure what are the values that can be supported in this param.
>
> https://github.com/apache/airflow/blob/a49224fa7ce45e9765c0d752edc30430e0d3ce14/scripts/ci/libraries/_initialization.sh#L306-L308
You can skip that part completely. It is going to be completely changed by in-progress "AIP-47 New design of System Tests" that we are working on now https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests - once we agree on it and implement it, this will simplify this part and the Breeze support for system tests will be removed (as it will not be needed any more)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1031444993
@potiuk Could you tell me what are the supported values for `ENABLED_SYSTEMS`? I could see that it's disabled by default. But I am not sure what are the values that can be supported in this param.
https://github.com/apache/airflow/blob/a49224fa7ce45e9765c0d752edc30430e0d3ce14/scripts/ci/libraries/_initialization.sh#L306-L308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038979622
@potiuk I could see that chmod permission is done in symbolic mode. Could this be done in octal mode? I have this doubt because when we do in symbolic mode we can toggle one of the group/user/other permission alone without touching others, but when we do in octal mode, it changes the entire permission. How to do this elegantly in pythonic way?
https://github.com/apache/airflow/blob/28378d867afaac497529bd2e1d2c878edf66f460/scripts/ci/libraries/_permissions.sh#L42-L53
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r805307010
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
I think so. The idea for the "verbose" part is to give the user a way to copy&paste the command and re-run it easily (maybe with small modifications) and to be able to diagnose potential bugs in setting the configuration in the CI.
So logging all variables there (i.e. being able to reproduce the exact command that caused the problem) is very important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r805291826
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pathlib import Path
+from typing import Dict, List
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
+ args_command = []
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ args_command.append("-e")
+ args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ args_command.append("-e")
+ args_command.append(constant_param_name + '=' + str(constant_param_value))
+ return args_command
+
+
Review comment:
sure @potiuk I will first solve the issue by fixing it in os.environ and then handle the verbose case too. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038210557
Still things pending in this task are:
- [ ] Fix file permission (need more clarity on this part)
- [ ] Building ci image instead of showing instructions to do it (which is currently in the code)
- [ ] Production image true specific changes. Currently only ci specific cases handled
- [ ] Adding trap
- [ ] Adding relevant test
If there is anything missing other than this part, please let me know @potiuk . Meanwhile, you can check the code, execute it and share your feedback too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038210557
Still things pending in this task are:
- [x] Fix file permission (need more clarity on this part)
- [x] Building ci image instead of showing instructions to do it (which is currently in the code)
- [ ] ~~Production image true specific changes. Currently only ci specific cases handled~~
- [ ] Adding trap
- [ ] Adding relevant test
If there is anything missing other than this part, please let me know @potiuk . Meanwhile, you can check the code, execute it and share your feedback too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r811294736
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
+ mock_console.print.assert_called_with('See https://docs.docker.com/compose/install/ for instructions. Make sure docker-compose you install is first on the PATH variable of yours.')
+
+
Review comment:
You can always do step-by-step-debugging in such case (this is where unit tests integrated with IDE are useful)
In this case the problem is that the regexp pattern does not match 2-segment version. It expects 3-segment one (1.29.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r811960847
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
Review comment:
@potiuk I have one more doubt in this mock part. we have two print statements if the docker-compose version is low. But in the mock, I could verify only one. Do you have any idea how I can add both prints in the mock?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r812635674
##########
File path: dev/breeze/tests/test_docker_command_utils.py
##########
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from unittest.mock import call
+
+from airflow_breeze.utils.docker_command_utils import check_docker_compose_version, check_docker_version
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_unknown(mock_console, mock_run_command):
+ check_docker_version(verbose=True)
+ expected_run_command_calls = [
+ call(['docker', 'info'], verbose=True, suppress_console_print=True, capture_output=True, text=True),
+ call(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ "Your version of docker is unknown. If the scripts fail, please make sure to"
+ " install docker at least: 20.10.0 version."
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_too_low(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "0.9"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with(
+ "Your version of docker is too old:0.9. Please upgrade to at least 20.10.0"
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_ok(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "20.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 20.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.check_docker_permission')
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_version_higher(mock_console, mock_run_command, mock_check_docker_permission):
+ mock_check_docker_permission.return_value = False
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "21.10.0"
+ check_docker_version(verbose=True)
+ mock_check_docker_permission.assert_called_with(True)
+ mock_run_command.assert_called_with(
+ ['docker', 'version', '--format', '{{.Client.Version}}'],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+ mock_console.print.assert_called_with("Good version of Docker: 21.10.0.")
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_unknown(mock_console, mock_run_command):
+ check_docker_compose_version(verbose=True)
+ expected_run_command_calls = [
+ call(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ ),
+ ]
+ mock_run_command.assert_has_calls(expected_run_command_calls)
+ mock_console.print.assert_called_with(
+ 'Unknown docker-compose version. At least 1.29 is needed! \
+ If Breeze fails upgrade to latest available docker-compose version'
+ )
+
+
+@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
+@mock.patch('airflow_breeze.utils.docker_command_utils.console')
+def test_check_docker_compose_version_low(mock_console, mock_run_command):
+ mock_run_command.return_value.returncode = 0
+ mock_run_command.return_value.stdout = "1.28.5"
+ check_docker_compose_version(verbose=True)
+ mock_run_command.assert_called_with(
+ ["docker-compose", "--version"],
+ verbose=True,
+ suppress_console_print=True,
+ capture_output=True,
+ text=True,
+ )
+
+ # mock_console.print.assert_called_with('You have too old version of docker-compose: 1.28.5! At least 1.29 is needed! Please upgrade!')
Review comment:
You can use `assert_has_calls` - you can find similar examples in airflow code :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1029192995
> @potiuk could you explain to me what this -n check does in this bash code?
Just check if they are set. If any of those are "net empty" then rather than using local sources of Airflow, we use airlfow either from:
* PYPI (if INSTALL_AIRFLOW_VERSION or USE_AIRFLOW_VERSION are set)
* GitHub (if INSTALL_AIRFLOW_REFERENCE is used.
Note that this is something that should be separated between PROD and CI image.
* USE_AIRFLOW_VERSION is for CI image
* INSTALL_AIRFLOW_REFERENCE and INSTALL_AIRFLOW_VERSION are for PROD image.
There is one difference between them : USE_AIRFLOW_VERSION will install airflow "after" you enter the image - so the "entrypoint_ci.sh" script will reinstall airflow in this case. Where INSTALL_AIRFLOW_VERSION or INSTALL_AIRFLOW_REFERENBCE are used in PROD image during the build phase - airlfow will be installed from PyPI/GiThub as part of one of the "docker/" scripts during the `docker build` command.
In our case we are still working on "build CI image" so the INSTALL* ones you can leave for later when we implement PROD image building.
> Also, I am quite confused about what variables should be in global_constants and what has to be in the dataclass.
That's simple. The global constants should contain the variables that do not change when parameters change. The dataclass should be used when you have a "calculated" value which depends on the parameters that you passed to the "breeze" command.
Actually that was one of the difficulty with Bash - because you can hardly distinguish those - env variable is an env variable and whether it is "unchangeable" or whether it has been calculated on-the-fly using passed parameters it is hard to tell. The Dataclass is our chance to distinguish between those two cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024887525
@potiuk Does `MSSQL_DATA_VOLUME` is created on a specific case? I tried running breeze in my machine and I couldn't see this folder getting created.
Also, could you explain to me the `CACHE_TMP_FILE_DIR=$(mktemp -d)` part? Why does it have to be done this way and what happens in add_traps? I could see we log info to OUTPUT_LOG. What's the best way to handle when migrating to python?
https://github.com/apache/airflow/blob/1d4b709e20b07c6f0b5d1bab1935e19557df2913/scripts/ci/libraries/_initialization.sh#L34-L82
My comment sounds like having too many questions :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024889471
Good questions are ALWAYS good. There are never too many of them :).
> @potiuk Does `MSSQL_DATA_VOLUME` is created on a specific case? I tried running breeze in my machine and I couldn't see this folder getting created.
They are created when `--backend mssql` is used - each backend keeps their database on a separate volume this way when you exit/enter breeze and restart the db, the data is kept across the container runs. The `breeze stop` command will remove all volumes on top of stopping the database, so it will cleanup everything. But if you do not `./breeze stop`, then the data is preserved in the DB,
> Also, could you explain to me the CACHE_TMP_FILE_DIR=$(mktemp -d) part? Why does it have to be done this way and what happens in add_traps? I could see we log info to OUTPUT_LOG. What's the best way to handle when migrating to python?
CACHE_TMP_FILE_DIR is used to store various temporary cached files:
* when we create diagrams automatically with `mermaid` (in one of the pre-commits)
* when we prepare md5 sum files (to compare them with the ones we store directly in .build directory)
Those files are usually not needed after we finish a breeze command - they are just used temporarily. "trap" makes sure that no matter if we nicely exist or Ctrl-C, or fail, those files are automatically deleted. The SKIP_CACHE_DELETION variable was just an escape hatch to not delete the files while debugging. I think we do not need it any more (and we can always coment out the code that deletes it when debugging).
In Python this "trap" functionality (not the same but good enough) can be achieved by https://docs.python.org/3/library/atexit.html. And I think this is the right way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024941042
> @potiuk Here irrespective of the backend chosen we create the file in cache. Am i missing to notice any specific code flow?
Not really. It coudl be protected by "if BACKEND == MSSQL" . It's just missing here :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025659012
You can read more about the bash magic you can do with variable substitution (this is the exact thing) https://tldp.org/LDP/abs/html/parameter-substitution.html
Also what's part of the magic here:
* `${1//[.-]/ }` - the 'space' between `/` and `}` matter - this is how we get the parameters to be separated
* It is important that there is no `"` around it. The `printf "%03d%03d%03d%.0s" "${1//[.-]/ }"` would not work because the "2 20 1" would be seen as single parameter. This is kinda unusual use and usually you want those quotes. This is why shellcheck (our pre-commit static check for bash code) complained and this is why I had to add ` # shellcheck disable=SC2086,SC2183 ` to stop it from complaining.
Now I think you should know why we want to get rid of Bash :). If explaining one line of bash takes 2-3 pages of text, it means that we should get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025707546
Some useful docs in semver library.
Version equality means for semver, that major, minor, patch, and prerelease parts are equal in both versions you compare. The build part is ignored
https://github.com/python-semver/python-semver/blob/master/docs/usage/determine-version-equality.rst
We can replace the prerelease part with empty string and do the comparison.
https://github.com/python-semver/python-semver/blob/master/docs/usage/replace-parts-of-a-version.rst
> However one of our users who run ./breeze in Azure Code QL ( which is Azure's CI system) got error that the version could not be parsed, because in Azure Code QL, they MODIFIED the version of docker they have installed and the version returned was something like:
>
> ```
> docker version --format '{{.Client.Version}}'
> 20.10.7+azure
> ```
Here the `+azure` would indicate the build part. So I think we don't need to remove it explicitely like how it's done using sed in bash. We can simply pass that to semver and it will work fine.
Please share your views on this @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r815429816
##########
File path: dev/breeze/src/airflow_breeze/breeze.py
##########
@@ -75,19 +111,78 @@ def version():
@option_verbose
-@main.command()
-def shell(verbose: bool):
+@main.command(
+ context_settings=dict(
+ ignore_unknown_options=True,
+ allow_extra_args=True,
+ ),
+)
+@option_python_version
+@option_backend
+@click.option('--integration', type=click.Choice(ALLOWED_INTEGRATIONS), multiple=True)
+@click.option('-L', '--build-cache-local', is_flag=True)
+@click.option('-U', '--build-cache-pulled', is_flag=True)
+@click.option('-X', '--build-cache-disabled', is_flag=True)
+@click.option('--production-image', is_flag=True)
Review comment:
@potiuk wrt [this](https://github.com/apache/airflow/pull/21848) issue, we don't need the `--production-image` flag and any process related to that flag, am I right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r815299052
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,199 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import sys
+from pathlib import Path
+from typing import Dict
+
+import click
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.ci.build_image import build_image
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ fix_group_permissions,
+ instruct_build_image,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_env_variables_docker_compose_command(shell_params: ShellBuilder) -> Dict[str, str]:
+ env_variables: Dict[str, str] = {}
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ env_variables[param_name] = str(getattr(shell_params, param_value))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ env_variables[constant_param_name] = str(constant_param_value)
+ return env_variables
+
+
+def build_image_checks(verbose: bool, shell_params: ShellBuilder):
+ fix_group_permissions()
+ build_ci_image_check_cache = Path(
+ BUILD_CACHE_DIR, shell_params.airflow_branch, f".built_{shell_params.python_version}"
+ )
+ if build_ci_image_check_cache.exists():
+ console.print(f'{shell_params.the_image_type} image already built locally.')
+ else:
+ console.print(f'{shell_params.the_image_type} image not built locally')
+ if not shell_params.force_build:
+ build_needed = md5sum_check_if_build_is_needed(
+ shell_params.md5sum_cache_dir, shell_params.the_image_type
+ )
+ if build_needed:
+ try:
Review comment:
Looks like this one is best: https://pypi.org/project/inputimeout/ (recommended here: https://stackoverflow.com/questions/1335507/keyboard-input-with-timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054281898
smth strange failed in build images. Let me close/reopen to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1050604172
@potiuk I could see that fix ownership is invoked only as part of PRODUCTION_IMAGE. Can you explain me why it has to be done that way?
https://github.com/apache/airflow/blob/608b8c4879c188881e057e6318a0a15f54c55c7b/breeze#L3475-L3484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r815433513
##########
File path: dev/breeze/src/airflow_breeze/breeze.py
##########
@@ -75,19 +111,78 @@ def version():
@option_verbose
-@main.command()
-def shell(verbose: bool):
+@main.command(
+ context_settings=dict(
+ ignore_unknown_options=True,
+ allow_extra_args=True,
+ ),
+)
+@option_python_version
+@option_backend
+@click.option('--integration', type=click.Choice(ALLOWED_INTEGRATIONS), multiple=True)
+@click.option('-L', '--build-cache-local', is_flag=True)
+@click.option('-U', '--build-cache-pulled', is_flag=True)
+@click.option('-X', '--build-cache-disabled', is_flag=True)
+@click.option('--production-image', is_flag=True)
Review comment:
BTW. You have now some exercise in conflict resolution as I merged the fix for storing python version :). Happy to help if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808904730
##########
File path: dev/breeze/src/airflow_breeze/utils/run_utils.py
##########
@@ -40,12 +41,22 @@ def run_command(
verbose: bool = False,
suppress_raise_exception: bool = False,
suppress_console_print: bool = False,
+ env: Optional[Mapping[str, str]] = None,
Review comment:
This is the "gist" and the problem you had been struggling with. Instead of modifying "os.environ" I pass the variables as dictionary to the "run_command".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808901770
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -117,18 +116,15 @@
}
-def construct_arguments_docker_compose_command(shell_params: ShellBuilder) -> List[str]:
- args_command = []
+def construct_env_variables_docker_compose_command(shell_params: ShellBuilder) -> Dict[str, str]:
+ env_variables: Dict[str, str] = {}
for param_name in PARAMS_TO_ENTER_SHELL:
param_value = PARAMS_TO_ENTER_SHELL[param_name]
- os.environ[param_name] = str(getattr(shell_params, param_value))
- args_command.append(param_name + '=' + str(getattr(shell_params, param_value)))
+ env_variables[param_name] = str(getattr(shell_params, param_value))
Review comment:
Here. Instead of modifying the "os.environ" I simply store the values in the dictionary and return iit.
Updating os.environ here is not really good because:
1) It modifies the env variables for whole process, not only for the command we want to run
2) It is a bit "unexpected side effect" of running the function. Generally functions like that should not modify any "global" state - they should at most take input and produce output (This is "functional" programming pattern).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r808903452
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -166,16 +162,14 @@ def build_image_checks(verbose: bool, shell_params: ShellBuilder):
check_docker_resources(verbose, str(shell_params.airflow_sources), shell_params.airflow_ci_image_name)
cmd = ['docker-compose', 'run', '--service-ports', '--rm', 'airflow']
cmd_added = shell_params.command_passed
+ env_variables = construct_env_variables_docker_compose_command(shell_params)
if cmd_added is not None:
cmd.extend(['-c', cmd_added])
if verbose:
- shell_params.print_badge_info
- show_cmd = construct_arguments_docker_compose_command(shell_params)
- show_cmd.extend(cmd)
- console.print('Command to run:', (' ').join(show_cmd))
- output = run_command(cmd, verbose=False, text=True)
+ shell_params.print_badge_info()
+ output = run_command(cmd, verbose=verbose, env=env_variables, text=True)
Review comment:
And here I pass the env variables to run command - this parameter was not present there - but I added it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#discussion_r815299089
##########
File path: dev/breeze/src/airflow_breeze/shell/enter_shell.py
##########
@@ -0,0 +1,199 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import sys
+from pathlib import Path
+from typing import Dict
+
+import click
+
+from airflow_breeze import global_constants
+from airflow_breeze.cache import (
+ check_cache_and_write_if_not_cached,
+ read_from_cache_file,
+ write_to_cache_file,
+)
+from airflow_breeze.ci.build_image import build_image
+from airflow_breeze.console import console
+from airflow_breeze.global_constants import (
+ FLOWER_HOST_PORT,
+ MSSQL_HOST_PORT,
+ MSSQL_VERSION,
+ MYSQL_HOST_PORT,
+ MYSQL_VERSION,
+ POSTGRES_HOST_PORT,
+ POSTGRES_VERSION,
+ REDIS_HOST_PORT,
+ SSH_PORT,
+ WEBSERVER_HOST_PORT,
+)
+from airflow_breeze.shell.shell_builder import ShellBuilder
+from airflow_breeze.utils.docker_command_utils import (
+ check_docker_compose_version,
+ check_docker_resources,
+ check_docker_version,
+)
+from airflow_breeze.utils.path_utils import BUILD_CACHE_DIR
+from airflow_breeze.utils.run_utils import (
+ filter_out_none,
+ fix_group_permissions,
+ instruct_build_image,
+ instruct_for_setup,
+ md5sum_check_if_build_is_needed,
+ run_command,
+)
+from airflow_breeze.visuals import ASCIIART, ASCIIART_STYLE, CHEATSHEET, CHEATSHEET_STYLE
+
+PARAMS_TO_ENTER_SHELL = {
+ "HOST_USER_ID": "host_user_id",
+ "HOST_GROUP_ID": "host_group_id",
+ "COMPOSE_FILE": "compose_files",
+ "PYTHON_MAJOR_MINOR_VERSION": "python_version",
+ "BACKEND": "backend",
+ "AIRFLOW_VERSION": "airflow_version",
+ "INSTALL_AIRFLOW_VERSION": "install_airflow_version",
+ "AIRFLOW_SOURCES": "airflow_sources",
+ "AIRFLOW_CI_IMAGE": "airflow_ci_image_name",
+ "AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_ci_image_name_with_tag",
+ "AIRFLOW_PROD_IMAGE": "airflow_prod_image_name",
+ "AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
+ "SQLITE_URL": "sqlite_url",
+ "USE_AIRFLOW_VERSION": "use_airflow_version",
+ "SKIP_TWINE_CHECK": "skip_twine_check",
+ "USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
+ "EXECUTOR": "executor",
+ "START_AIRFLOW": "start_airflow",
+ "ENABLED_INTEGRATIONS": "enabled_integrations",
+ "GITHUB_ACTIONS": "github_actions",
+ "ISSUE_ID": "issue_id",
+ "NUM_RUNS": "num_runs",
+ "VERSION_SUFFIX_FOR_SVN": "version_suffix_for_svn",
+ "VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
+}
+
+PARAMS_FOR_SHELL_CONSTANTS = {
+ "SSH_PORT": SSH_PORT,
+ "WEBSERVER_HOST_PORT": WEBSERVER_HOST_PORT,
+ "FLOWER_HOST_PORT": FLOWER_HOST_PORT,
+ "REDIS_HOST_PORT": REDIS_HOST_PORT,
+ "MYSQL_HOST_PORT": MYSQL_HOST_PORT,
+ "MYSQL_VERSION": MYSQL_VERSION,
+ "MSSQL_HOST_PORT": MSSQL_HOST_PORT,
+ "MSSQL_VERSION": MSSQL_VERSION,
+ "POSTGRES_HOST_PORT": POSTGRES_HOST_PORT,
+ "POSTGRES_VERSION": POSTGRES_VERSION,
+}
+
+PARAMS_IN_CACHE = {
+ 'python_version': 'PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'BACKEND',
+ 'executor': 'EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+DEFAULT_VALUES_FOR_PARAM = {
+ 'python_version': 'DEFAULT_PYTHON_MAJOR_MINOR_VERSION',
+ 'backend': 'DEFAULT_BACKEND',
+ 'executor': 'DEFAULT_EXECUTOR',
+ 'postgres_version': 'POSTGRES_VERSION',
+ 'mysql_version': 'MYSQL_VERSION',
+ 'mssql_version': 'MSSQL_VERSION',
+}
+
+
+def construct_env_variables_docker_compose_command(shell_params: ShellBuilder) -> Dict[str, str]:
+ env_variables: Dict[str, str] = {}
+ for param_name in PARAMS_TO_ENTER_SHELL:
+ param_value = PARAMS_TO_ENTER_SHELL[param_name]
+ env_variables[param_name] = str(getattr(shell_params, param_value))
+ for constant_param_name in PARAMS_FOR_SHELL_CONSTANTS:
+ constant_param_value = PARAMS_FOR_SHELL_CONSTANTS[constant_param_name]
+ env_variables[constant_param_name] = str(constant_param_value)
+ return env_variables
+
+
+def build_image_checks(verbose: bool, shell_params: ShellBuilder):
+ fix_group_permissions()
+ build_ci_image_check_cache = Path(
+ BUILD_CACHE_DIR, shell_params.airflow_branch, f".built_{shell_params.python_version}"
+ )
+ if build_ci_image_check_cache.exists():
+ console.print(f'{shell_params.the_image_type} image already built locally.')
+ else:
+ console.print(f'{shell_params.the_image_type} image not built locally')
+ if not shell_params.force_build:
+ build_needed = md5sum_check_if_build_is_needed(
+ shell_params.md5sum_cache_dir, shell_params.the_image_type
+ )
+ if build_needed:
+ try:
Review comment:
Just add it to setup.cfg in dev/breeze and you will be able to use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1047052751
@potiuk One of the tests that I tried writing with mock patch was failing. I am not sure where I am going wrong.
```
@mock.patch('airflow_breeze.utils.docker_command_utils.run_command')
@mock.patch('airflow_breeze.utils.docker_command_utils.console')
def test_check_docker_compose_version_ok(mock_console, mock_run_command):
mock_run_command.return_value.returncode = 0
mock_run_command.return_value.stdout = "1.29"
check_docker_compose_version(verbose=True)
mock_run_command.assert_called_with(
["docker-compose", "--version"],
verbose=True,
suppress_console_print=True,
capture_output=True,
text=True,
)
> mock_console.print.assert_called_with("Good version of docker-compose: 1.29")
dev/breeze/tests/test_docker_command_utils.py:156:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_mock_self = <MagicMock name='console.print' id='13971730345[96](https://github.com/apache/airflow/runs/5277010738?check_suite_focus=true#step:5:96)00'>
args = ('Good version of docker-compose: 1.29',), kwargs = {}
expected = "print('Good version of docker-compose: 1.29')"
def assert_called_with(_mock_self, *args, **kwargs):
"""assert that the mock was called with the specified arguments.
Raises an AssertionError if the args and keyword args passed in are
different to the last call to the mock."""
self = _mock_self
if self.call_args is None:
expected = self._format_mock_call_signature(args, kwargs)
> raise AssertionError('Expected call: %s\nNot called' % (expected,))
E AssertionError: Expected call: print('Good version of docker-compose: 1.29')
E Not called
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/unittest/mock.py:869: AssertionError
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054550160
@potiuk I have some issue in resolving conflicts. It shows only one conflict, but when i resolve and push the code, many of the changes that i made goes missing
Edit: I have fixed it. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1054497600
@potiuk I have cleaned up the unused variables and commented one's. You can verify it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025659012
You can read more about the bash magic you can do with variable substitution (this is the exact name of what we do here) https://tldp.org/LDP/abs/html/parameter-substitution.html
Also what's part of the magic here:
* `${1//[.-]/ }` - the 'space' between `/` and `}` matter - this is how we get the parameters to be separated - basically we replace any "." or "-" with space with this construct. The `//` means that this is "global" replacement - so it will replace "all" . and - found not only the first (`${1/[.-]/ }` only replaces the first it finds).
* It is important that there is no `"` around it. The `printf "%03d%03d%03d%.0s" "${1//[.-]/ }"` would not work because the "2 20 1" would be seen as a single parameter, but if you do not add `"` the space we added in the previous step, makes each part of the number a separate parameter to `printf`. This is kinda unusual use and usually you want those quotes. This is why shellcheck (our pre-commit static check for bash code) complained and this is why I had to add ` # shellcheck disable=SC2086,SC2183 ` to stop it from complaining.
Now I think you should know why we want to get rid of Bash :). If explaining one line of bash takes 2-3 paragraph of text, it means that we should get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025659012
You can read more about the bash magic you can do with variable substitution (this is the exact thing) https://tldp.org/LDP/abs/html/parameter-substitution.html
Also what's part of the magic here:
* `${1//[.-]/ }` - the 'space' between `/` and `}` matter - this is how we get the parameters to be separated - basically we replace any "." or "-" with space with this construct.
* It is important that there is no `"` around it. The `printf "%03d%03d%03d%.0s" "${1//[.-]/ }"` would not work because the "2 20 1" would be seen as a single parameter, but if you do not add `"` the space we added in the previous step, makes each part of the number a separate parameter to `printf`. This is kinda unusual use and usually you want those quotes. This is why shellcheck (our pre-commit static check for bash code) complained and this is why I had to add ` # shellcheck disable=SC2086,SC2183 ` to stop it from complaining.
Now I think you should know why we want to get rid of Bash :). If explaining one line of bash takes 2-3 paragraph of text, it means that we should get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025089492
> Do we have to use the bash commands you have used in lines 924 and 935 via subcommand in python? Or do we have to try a different way? Also, I don't understand the comment added `In GitHub Code QL, the version of docker has +azure suffix which we should remove`. Could you explain to me that part?
Yeah. We should run the `docker commands` via subprocess. We **could** potentially use docker API in python, but running docker as `docker` command is precisely what we want to check . In this check we want to check if the user has:
* good docker command line version and it is on the PATH
* the version is properly configured - when you install docker on Linux you have to perform [additional steps](https://docs.docker.com/engine/install/linux-postinstall/) to make docker available for you user - the "permission" error is the way to check it (however it would be great to see if we have the message right, it could have changed). The way to check it is to remove yourself from docker group or run docker as a diferent user who is not in `docker` group
The best way to check it is to actually run the commands :)
> In GitHub Code QL, the version of docker has +azure suffix which we should remove.
The version check simply checks the min major/minor version and in Python it can be written in a nicer way. The point here is that usualy the version printed by Docker is like that:
```
docker version --format '{{.Client.Version}}'
20.10.7
```
However one of our users who run ./breeze in Azure Code QL ( which is Azure's CI system) got error that the version could not be parsed, because in Azure Code QL, they MODIFIED the version of docker they have installed and the version returned was something like:
```
docker version --format '{{.Client.Version}}'
20.10.7+azure
```
:scream: :scream: :scream: :scream: :scream:
So our version check must include also this case - when parsing the version .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1025647705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1023189760
> @potiuk I want to know which is the correct set of supported Postgres versions.
Both :). The first are "All possible" the second is "all that run in CI" - we decided to only run lowest and highest in Postgres as we know Postgres is stable (so if 10 an 13 works, we are pretty sure 11 and 12 will work too).
In this cae we should use the 10, 11, 12, 13 and we can automatically infer lowest and highest in the "selective check" script which is used in CI to determine which is the matrix of tests to run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1024889471
Good questions are ALWAYS good. There are never too many of them :).
> @potiuk Does `MSSQL_DATA_VOLUME` is created on a specific case? I tried running breeze in my machine and I couldn't see this folder getting created.
They are created when `--backend mssql` is used - each backend keeps their database on a separate volume this way when you exit/enter breeze and restart the db, the data is kept across the container runs. The `breeze stop` command will remove all volumes on top of stopping the database, so it will cleanup everything. But if you do not `./breeze stop`, then the data is preserved in the DB,
> Also, could you explain to me the CACHE_TMP_FILE_DIR=$(mktemp -d) part? Why does it have to be done this way and what happens in add_traps? I could see we log info to OUTPUT_LOG. What's the best way to handle when migrating to python?
CACHE_TMP_FILE_DIR is used to store various temporary cached files:
* when we create diagrams automatically with `mermaid` (in one of the pre-commits)
* when we prepare md5 sum files (to compare them with the ones we store directly in .build directory)
Those files are usually not needed after we finish a breeze command - they are just used temporarily. "trap" makes sure that no matter if we nicely exist or Ctrl-C, or fail, those files are automatically deleted. The SKIP_CACHE_DELETION variable was just an escape hatch to not delete the files while debugging. I think we do not need it any more (and we can always coment out the code that deletes it when debugging).
In Python this "trap" functionality (not the same but good enough) can be achieved by https://docs.python.org/3/library/atexit.html. And I think this is the right way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1029581924
> That's simple. The global constants should contain the variables that do not change when parameters change. The dataclass should be used when you have a "calculated" value which depends on the parameters that you passed to the "breeze" command.
> Actually that was one of the difficulty with Bash - because you can hardly distinguish those - env variable is an env variable and whether it is "unchangeable" or whether it has been calculated on-the-fly using passed parameters it is hard to tell. The Dataclass is our chance to distinguish between those two cases.
yes, I could understand this part, but figuring out if the variable remains constant / varies depending on 'parameters' passed to breeze command is the part where I am confused. Say example `USE_AIRFLOW_VERSION ` does it have to go to global constants or dataclass. Does the user can set this value / does it have to remain constant? How do I figure out that? @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #21145:
URL: https://github.com/apache/airflow/pull/21145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1035006093
> @potiuk Am I right in understanding that I have to show message/instruction asking user to do build part? Or do i have to execute the `build-ci-image`/`build-prod-image` command?
Execute the command. The flow should be like that:
1) User runs ./breeze without `--force-build`:
* if the image is "fresh" - just start breeze shell
* if the image is "not-fresh" - display the warning and give the user 4 seconds to answer "y/n/q" question whether to build image
y -> build
n -> continue without building
q -> quit
2) User runs ./breeze with `--force-build`:
* no matter what the "build is started"
There is just one thing to watch out: when we start the build in either of the scenarios above, we start an addtional check. We check if the user is rebased to latest "main" (there is a piece of code in the current breeze already).
If the user is not rebased, then we print the warning "This will take a lot of time, we think you should rebase first, but if you really, really want - you can do it". Here "y/n/q" question again - but this time wihout "timeout". The users should deliberately press "y" if they try to rebuild, but they are not rebased. This might lead to really long build times so it should be really confirmed by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1038210557
Still things pending in this task are:
- [ ] Fix file permission (need more clarity on this part)
- [ ] Building ci image instead of showing instructions to do it (which is currently in the code)
- [x] Production image true specific changes. Currently only ci specific cases handled
- [ ] Adding trap
- [ ] Adding relevant test
If there is anything missing other than this part, please let me know @potiuk . Meanwhile, you can check the code, execute it and share your feedback too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039144587
> @potiuk I am not very clear about this part of the code too. How does this command helps to find the deleted files.
You can do it in python by simply checking if a file exist before attempting to run chmod.
This one is "bash-y" way of handling the case that you had a file in git, but you just deleted it in your commit. In this case it will be present in the `git ls-files` output, but it will not be physically there so running "chmod" will fail :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1039144587
> @potiuk I am not very clear about this part of the code too. How does this command helps to find the deleted files.
You can do it in python by simply trying if a file exist before attempting to run chmod.
This one is "bash-y" way of handling the case that you had a file in git, but you just deleted it in your commit. In this case it will be present in the `git ls-files` output, but it will not be physically there so running "chmod" will fail :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1037259920
@potiuk I am trying to run docker-compose command via subprocess and I face error when I pass env variables as -e. It works properly when set as os.environ variable and executing the docker-compose command. But it didn't work that way when I tried via `-e` flag. I will check if I could fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21145:
URL: https://github.com/apache/airflow/pull/21145#issuecomment-1037260947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org