You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/07 08:57:15 UTC
[GitHub] [airflow] potiuk opened a new issue #20741: CI: Reuse Breeze CI image building for CI
potiuk opened a new issue #20741:
URL: https://github.com/apache/airflow/issues/20741
We already have a basic Build image function in `Breeze` - we should be able to use it to build images in CI.
There are actions that build images in `./build-imaages.yml` and they should simply (similarely to free-space) use Python commands develped in ./dev/breeze.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1058149691
I found that the generated image is wrong, nothing inside, 0B.
REPOSITORY TAG IMAGE ID CREATED SIZE
ghcr.io/edithturn/airflow/main/ci/python3.9 eda6a85304bb2b598674a94e1b877a920e1451db 71de1148337f N/A 0B
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1048098325
@potiuk I have a doubt about the construction of the image.
The image we are building for build_image is Dockerfile.ci, and this creates the URL based in ARG PYTHON_BASE_IMAGE="python:3.7-slim-buster" but seeing your, the name of the image is different, without slim-buster (see picture).
Is this the URL of the basic image that we take to build that Dockerfil?
I am getting a: denied, is this because of the malformation of the URL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1058131956
woho, it passed the build-ci-image, now I am checking the problem with Push
But I think it is weird because I am not taking much than one env variable for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1046570768
For example `PYTHON_MAJOR_MINOR_VERSION` variables (which is set from matrix in the CI) should be used as fallback in the `--python` flag in `build-ci-image`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1048540404
Few comments:
* use `Breeze2` instead of `./Breeze2` - similarly as `airflow-freespace`, Breeze2 is already installed by the earlier `pip install` step
* You will need to modify the options of build-ci-image command to use ENV_VARIABLES. In the original Breeze, we read the variables during intialization, but in Python command we just need to make sure the options can be read from the env variable. See this chapter in click docs: https://click.palletsprojects.com/en/8.0.x/options/#values-from-environment-variables
You will need to modify this code: https://github.com/apache/airflow/blob/e8afff9cb52c8c2bc8bd6587656c67703fd87956/dev/breeze/src/airflow_breeze/breeze.py#L93
For example:
```
@click.option('-p', '--python', help='Choose your python version')
```
You will have to change to:
```
@click.option('-p', '--python', help='Choose your python version', envvar='PYTHON_MAJOR_MINOR_VERSION')
```
If you look at the execution of the build step in CI you will see that there are already many variables set there by ci,yaml and selective checks.
```
env:
MOUNT_SELECTED_LOCAL_SOURCES: false
FORCE_ANSWER_TO_QUESTIONS: yes
CHECK_IMAGE_FOR_REBUILD: true
SKIP_CHECK_REMOTE_IMAGE: true
DB_RESET: true
VERBOSE: true
GITHUB_REPOSITORY: edithturn/airflow
GITHUB_USERNAME: edithturn
CONSTRAINTS_GITHUB_REPOSITORY: apache/airflow
GITHUB_REGISTRY_PULL_IMAGE_TAG: latest
INSTALL_PROVIDERS_FROM_SOURCES: true
AIRFLOW_LOGIN_TO_GITHUB_REGISTRY: true
BACKEND: postgres
PYTHON_MAJOR_MINOR_VERSION: 3.7
UPGRADE_TO_NEWER_DEPENDENCIES: false
DOCKER_CACHE: disable:
```
For example the case you mentioned is handled by `GITHUB_REPOSITORY` variable.
You will find that there is this option:
```
@click.option('--github-repository', help='Choose repository to push/pull image.')
```
And again adding `envvar='GITHUB_REPOSITORY'`) should help.
You will find the variables used for each option in https://github.com/apache/airflow/blob/main/breeze#L799 - there is "parse-flags" and each flag has appropriate env variable assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1058131956
woho, it passed the build-ci-image, now I am checking the problem with Push
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1046569543
> Understoonn @potiuk, I was testing and the image is not being built appropriately. I just print a `docker images`
Actually I think now it will be better to just use "Breeze2 build-ci-image" command directly - no need for a separate entrypoint. I think you should be able to run the command even now - as "Breeze2" entrypoint is already installed at this point.
> Another doubt I have. What should be the parameter for build_image, since in the build_image this recipe: **kwargs as an argument, it means many arguments, like the python version, directories, etc. How I could know what will be these parameters?
Those parameters should be (as the yare now) passed through environment variables. When you run it as `./Breeze2 build-ci-image` you should just make sure that al the necessary variables that are currently set in the build-images.yml are properly configured as proper click options env variables fallbacks https://click.palletsprojects.com/en/8.0.x/options/#values-from-environment-variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1048098325
@potiuk I have a doubt about this part.
The image we are building for build_image is Dockerfile.ci, and this creates the URL based in ARG PYTHON_BASE_IMAGE="python:3.7-slim-buster" but seeing your, the name of the image is different, without slim-buster (see picture).
Is this the URL of the basic image that we take to build that Dockerfil?
I am getting a: denied, is this because of the malformation of the URL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1054511731
@potiuk I was struggling with a problem to know why the `.build` folder was not created when executing build_image. It is that `.build ` is only created in the breeze main. Really sorry for the time it took me to realize it.
https://github.com/apache/airflow/blob/5d89dea56843d7b76d5e308e373ba16ecbcffa77/dev/breeze/src/airflow_breeze/breeze.py#L358-L360
So what I did was copy the line to build_image:
https://github.com/edithturn/airflow/blob/e282d37962393d76802ec8170b6c0d4cef290f20/dev/breeze/src/airflow_breeze/breeze.py#L166-L169
With this, I am not having the problem "file not found", so I could start to add the other variable environments that I need to build the image.
As far I know, we don't run breeze in the Ci GitHub Actions, so these breeze main where that directory is created never will execute it never will create that directory which is needed to build the image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1045259939
@potiuk I changed the file ci_build_ci_image_on_ci.sh for build_image.py, it is the unique part where we call to build the image. Please let me know if I am in the right way.
https://github.com/edithturn/airflow/blob/a02e8562a01b251d214f06f69bf1f15c91af536c/.github/workflows/build-images.yml#L231-L235
Another thing I was thinking is If I finished building the image, it means that it is ready to push and It is made for ci_push_ci_images.sh, and it makes a login in docker registry to push the image. Will, it is possible to test it in my docker registry with my own credentials? How will I prove that the image is well built and ready to upload?
Note: I am testing it in my own remote since these are changes made in the build-image.yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1048540404
Few comments:
* use `Breeze2` instead of `./Breeze2` - similarly as `airflow-freespace`, Breeze2 is already installed by the earlier `pip install` step
* You will need to modify the options of build-ci-image command to use ENV_VARIABLES. In the original Breeze, we read the variables during intialization, but in Python command we just need to make sure the options can be read from the env variable. See this chapter in click docs: https://click.palletsprojects.com/en/8.0.x/options/#values-from-environment-variables
You will need to modify this code: https://github.com/apache/airflow/blob/e8afff9cb52c8c2bc8bd6587656c67703fd87956/dev/breeze/src/airflow_breeze/breeze.py#L93
For example:
```
@click.option('-p', '--python', help='Choose your python version')
```
You will have to change to:
```
@click.option('-p', '--python', help='Choose your python version', envvar='PYTHON_MAJOR_MINOR_VERSION')
```
If you look at the execution of the build step in CI you will see that there are already many variables set there by ci,yaml and selective checks.
```
env:
MOUNT_SELECTED_LOCAL_SOURCES: false
FORCE_ANSWER_TO_QUESTIONS: yes
CHECK_IMAGE_FOR_REBUILD: true
SKIP_CHECK_REMOTE_IMAGE: true
DB_RESET: true
VERBOSE: true
GITHUB_REPOSITORY: edithturn/airflow
GITHUB_USERNAME: edithturn
CONSTRAINTS_GITHUB_REPOSITORY: apache/airflow
GITHUB_REGISTRY_PULL_IMAGE_TAG: latest
INSTALL_PROVIDERS_FROM_SOURCES: true
AIRFLOW_LOGIN_TO_GITHUB_REGISTRY: true
BACKEND: postgres
PYTHON_MAJOR_MINOR_VERSION: 3.7
UPGRADE_TO_NEWER_DEPENDENCIES: false
DOCKER_CACHE: disable:
```
For example the case you mentioned is handled by `GITHUB_REPOSITORY` variable.
You will find that there is this option:
```
@click.option('--github-repository', help='Choose repository to push/pull image.')
```
And again adding `envvar='GITHUB_REPOSITORY`) should help.
You will find the variables used for each option in https://github.com/apache/airflow/blob/main/breeze#L799 - there is "parse-flags" and each flag has appropriate env variable assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1046569543
> Understoonn @potiuk, I was testing and the image is not being built appropriately. I just print a `docker images`
Actually I think now it will be better to just use "Breeze2 build-ci-image" command directly - no need for a separate entrypoint. I think you should be able to run the command even now - as "Breeze2" entrypoint is already installed at this point.
> Another doubt I have. What should be the parameter for build_image, since in the build_image this recipe: **kwargs as an argument, it means many arguments, like the python version, directories, etc. How I could know what will be these parameters?
Those parameters should be (as they are now) passed through environment variables. When you run it as `./Breeze2 build-ci-image` you should just make sure that al the necessary variables that are currently set in the build-images.yml are properly configured as proper click options env variables fallbacks https://click.palletsprojects.com/en/8.0.x/options/#values-from-environment-variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1055551641
> It should not fail when it is random number (I saw it in one of your executions: https://github.com/potiuk/airflow/runs/5344689690?check_suite_focus=true), so I have to find a way to continue the process if this value is random, right? I must be setting the variable to boolean yes or yes somewhere, and that is why it fails.
AAARGH. The "Upgrade to newer depencies" is rather special. It can be "true", "false" OR ANYTHING_ELSE :). Simply change the type of the parameter to "string". We might want to improve it in the future, but for now it should be fine to just change it to "string" and possibly add some description for that one in "Help"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1046297844
Understoonn @potiuk, I was testing and the image is not being built appropriately. I just print a `docker images`
I saw that in breeze.py the build_image is being used:
https://github.com/apache/airflow/blob/9ad4de835cbbd296b9dbd1ff0ea88c1cd0050263/dev/breeze/src/airflow_breeze/breeze.py#L162-L164
Could it be a good example to see if the buid_image is working effectible?
Another point should be the parameter for build_image, since this has: **kwargs as an argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1055548139
> As far I know, we don't run breeze in the Ci GitHub Actions, so these breeze main where that directory is created never will execute it never will create that directory which is needed to build the image.
Hmm. I think this will be fixed when #21145 is merged - for now this is good to keep it but once we merge the change from @bowrna we should remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1055043517
@potiuk another thing I found is with the env variable **UPGRADE_TO_NEWER_DEPENDENCIES**, it is setting a random number.
 [JOB LOG](https://github.com/edithturn/airflow/runs/5370490394?check_suite_focus=true)
It is because I am **pushing** the code and, It falls into this validation.
https://github.com/apache/airflow/blob/82e1abf7144f5386cb9dbf9775e9c78da51ec673/scripts/ci/selective_ci_checks.sh#L45-L54
It should not fail when it is random number (I saw it in one of your executions: https://github.com/potiuk/airflow/runs/5344689690?check_suite_focus=true), so I have to find a way to continue the process if this value is random, right? I must be setting the variable to boolean yes or yes somewhere, and that is why it fails.
Could you please confirm that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1058209331
@potiuk could you please assign me this task? 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1045882611
> @potiuk I changed the file ci_build_ci_image_on_ci.sh for build_image.py, it is the unique part where we call to build the image. Please let me know if I am in the right way.
>
> https://github.com/edithturn/airflow/blob/a02e8562a01b251d214f06f69bf1f15c91af536c/.github/workflows/build-images.yml#L231-L235
Yeah. But we also should make it an entrypoint similar to "airflow-freespace", (`airlfow-build-image`) this way when we run it in CI, we shoudl be able to - similarly as `airflow-freespace` run it as `run: airflow-build-image`.
> Another thing I was thinking is If I finished building the image, it means that it is ready to push and It is made for ci_push_ci_images.sh, and it makes a login in docker registry to push the image. Will, it is possible to test it in my docker registry with my own credentials? How will I prove that the image is well built and ready to upload?
Yeah. It should work in your repo. That should be no problem. If everything works fine with the build image and all the variables are properly used, the image should be built as:
```
ghcr.io/edithurn/airflow/main/ci/python3.7
```
GIHUB_REPOSITORY variable should be used to determine that.
See for example here: https://github.com/potiuk/airflow/runs/5177272408?check_suite_focus=true
```
Build CI image ghcr.io/potiuk/airflow/main/ci/python3.7
```
Of course there might be some issues with the python build_image.py to use that variable (we did not try it before) - but then this is something that should be fixed if it will continue using "apache/airflow" :).
> Note: I am testing it in my own remote since these are changes made in the build-image.yaml
Yep. The only way to test it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1046297844
Understoonn @potiuk, I was testing and the image is not being built appropriately. I just print a `docker images`
I saw that in breeze.py the build_image is being used:
https://github.com/apache/airflow/blob/9ad4de835cbbd296b9dbd1ff0ea88c1cd0050263/dev/breeze/src/airflow_breeze/breeze.py#L162-L164
Could it be a good example to see if the buid_image is working effectible?
Another doubt I have. What should be the parameter for build_image, since in the build_image this recipe: **kwargs as an argument, it means many arguments, like the python version, directories, etc. How I could know what will be these parameters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1048540404
Few comments:
* use `Breeze2` instead of `./Breeze2` - similarly as `airflow-freespace`, Breeze2 is already installed by the earlier `pip install` step
* You will need to modify the options of build-ci-image command to use ENV_VARIABLES. in the original Breeze, we read the variables during intialization, but in Python command we just need to make sure the options can be read from the env variable. See this chapter in click docs: https://click.palletsprojects.com/en/8.0.x/options/#values-from-environment-variables
You will need to modify this code: https://github.com/apache/airflow/blob/e8afff9cb52c8c2bc8bd6587656c67703fd87956/dev/breeze/src/airflow_breeze/breeze.py#L93
For example:
```
@click.option('-p', '--python', help='Choose your python version')
```
You will have to change to:
```
@click.option('-p', '--python', help='Choose your python version', envvar='PYTHON_MAJOR_MINOR_VERSION')
```
If you look at the execution of the build step in CI you will see that there are already many variables set there by ci,yaml and selective checks.
```
env:
MOUNT_SELECTED_LOCAL_SOURCES: false
FORCE_ANSWER_TO_QUESTIONS: yes
CHECK_IMAGE_FOR_REBUILD: true
SKIP_CHECK_REMOTE_IMAGE: true
DB_RESET: true
VERBOSE: true
GITHUB_REPOSITORY: edithturn/airflow
GITHUB_USERNAME: edithturn
CONSTRAINTS_GITHUB_REPOSITORY: apache/airflow
GITHUB_REGISTRY_PULL_IMAGE_TAG: latest
INSTALL_PROVIDERS_FROM_SOURCES: true
AIRFLOW_LOGIN_TO_GITHUB_REGISTRY: true
BACKEND: postgres
PYTHON_MAJOR_MINOR_VERSION: 3.7
UPGRADE_TO_NEWER_DEPENDENCIES: false
DOCKER_CACHE: disable:
```
For example the case you mentioned is handled by `GITHUB_REPOSITORY` variable.
You will find that there is this option:
```
@click.option('--github-repository', help='Choose repository to push/pull image.')
```
And again adding `envvar='GITHUB_REPOSITORY`) should help.
You will find the variables used for each option in https://github.com/apache/airflow/blob/main/breeze#L799 - there is "parse-flags" and each flag has appropriate env variable assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1055854032
Thanks Jarek, It worked setting it as a string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20741:
URL: https://github.com/apache/airflow/issues/20741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1046569543
> Understoonn @potiuk, I was testing and the image is not being built appropriately. I just print a `docker images`
Actually I think now it will be better to just use "Breeze2 build-ci-image" command directly - no need for a separate entrypoint. I think you should be able to run the command even now - as "Breeze2" entrypoint is already installed at this point.
> Another doubt I have. What should be the parameter for build_image, since in the build_image this recipe: **kwargs as an argument, it means many arguments, like the python version, directories, etc. How I could know what will be these parameters?
Those parameters should be (as they are now) passed through environment variables. When you run it as `./Breeze2 build-ci-image` you should just make sure that al the necessary variables that are currently set in the `build-images.yml` are properly configured as proper click options env variables fallbacks https://click.palletsprojects.com/en/8.0.x/options/#values-from-environment-variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1054511731
@potiuk I was struggling with a problem to know why the `.build` folder was not created when executing build_image. It is that `.build ` is only created in the breeze main. Really sorry for the time it took me to realize it.
https://github.com/apache/airflow/blob/5d89dea56843d7b76d5e308e373ba16ecbcffa77/dev/breeze/src/airflow_breeze/breeze.py#L358-L360
So what I did was copy the line to:
https://github.com/edithturn/airflow/blob/e282d37962393d76802ec8170b6c0d4cef290f20/dev/breeze/src/airflow_breeze/breeze.py#L166-L169
With this, I am not having the problem "file not found", so I could start to add the other variable environments that I need to build the image.
As far I know, we don't run breeze in the Ci GitHub Actions, so these breeze main where that directory is created never will execute it never will create that directory which is needed to build the image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1054511731
@potiuk I was struggling with a problem to know why the `.build` folder was not created when executing build_image. It is that `.build ` is only created in the breeze main. Really sorry for the time it took me to realize it.
https://github.com/edithturn/airflow/blob/e282d37962393d76802ec8170b6c0d4cef290f20/dev/breeze/src/airflow_breeze/breeze.py#L361-L363
So what I did was copy the line to:
https://github.com/edithturn/airflow/blob/e282d37962393d76802ec8170b6c0d4cef290f20/dev/breeze/src/airflow_breeze/breeze.py#L166-L169
With this, I am not having the problem "file not found", so I could start to add the other variable environments that I need to build the image.
As far I know, we don't run breeze in the Ci GitHub Actions, so these breeze main where that directory is created never will execute it never will create that directory which is needed to build the image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20741: CI: Reuse Breeze CI image building for CI
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20741:
URL: https://github.com/apache/airflow/issues/20741#issuecomment-1055043517
@potiuk another thing I found is with the env variable **UPGRADE_TO_NEWER_DEPENDENCIES**, it is setting a random number.
 [job](https://github.com/edithturn/airflow/runs/5370490394?check_suite_focus=true)
It is because I am **pushing** the code and, It falls into this validation.
https://github.com/apache/airflow/blob/82e1abf7144f5386cb9dbf9775e9c78da51ec673/scripts/ci/selective_ci_checks.sh#L45-L54
It should not fail when it is random number (I saw it in one of your executions: https://github.com/potiuk/airflow/runs/5344689690?check_suite_focus=true), so I have to find a way to continue the process if this value is random, right? I must be setting the variable to boolean yes or yes somewhere, and that is why it fails.
Could you please confirm that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org