You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Lee-W (via GitHub)" <gi...@apache.org> on 2023/11/23 11:19:53 UTC
[PR] add guideline for missing image when running breeze [airflow]
Lee-W opened a new pull request, #35813:
URL: https://github.com/apache/airflow/pull/35813
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #35813:
URL: https://github.com/apache/airflow/pull/35813#discussion_r1403255497
##########
BREEZE.rst:
##########
@@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved
The similar issue could happen if you are behind an HTTP/HTTPS proxy and your access to required websites are
blocked by it, or your proxy setting has not been done properly.
+"No such image" Error:
+------------------------
+
+When running ``breeze start-airflow`` or commands that needs a base image, the following output might be observed:
+
+.. code-block:: bash
+
+ [+] Creating 3/0
+ ✔ Container docker-compose-rabbitmq-1 Running 0.0s
+ ✔ Container docker-compose-redis-1 Running 0.0s
+ ✔ Container docker-compose-postgres-1 Running 0.0s
+ [+] Building 0.0s (0/0) docker:desktop-linux
+ Error response from daemon: No such image: ghcr.io/apache/airflow/main/ci/python3.10:latest
+ Error 1 returned
+
+In this case, we'll need to pull the missing image ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
Review Comment:
In this case, we'll need to pull the missing image. For example, for Python 3.10 we need to use ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1826801193
The part that fixes the problem you had is here:
https://github.com/apache/airflow/pull/35862/files#diff-6ef5b756e45d03d3723ba4a5c4b0f1417b115fe25560f293351c30290ec67b31R127
```
"""Starts building attempt. Returns false if we should not continue"""
result = run_command(
["docker", "inspect", ci_image_params.airflow_image_name_with_tag],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
return True
if ci_image_params.skip_image_upgrade_check:
return False
if not ci_image_params.force_build and not ci_image_params.upgrade_to_newer_dependencies:
if not should_we_run_the_build(build_ci_params=ci_image_params):
return False
return True
```
But it's done together with renaming of e "image_check" property in the object - that's why it's easier to keep it together with the rename.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #35813:
URL: https://github.com/apache/airflow/pull/35813#discussion_r1403253983
##########
BREEZE.rst:
##########
@@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved
The similar issue could happen if you are behind an HTTP/HTTPS proxy and your access to required websites are
blocked by it, or your proxy setting has not been done properly.
+"No such image" Error:
+------------------------
+
+When running ``breeze start-airflow`` or commands that needs a base image, the following output might be observed:
+
+.. code-block:: bash
+
+ [+] Creating 3/0
+ ✔ Container docker-compose-rabbitmq-1 Running 0.0s
+ ✔ Container docker-compose-redis-1 Running 0.0s
+ ✔ Container docker-compose-postgres-1 Running 0.0s
+ [+] Building 0.0s (0/0) docker:desktop-linux
+ Error response from daemon: No such image: ghcr.io/apache/airflow/main/ci/python3.10:latest
+ Error 1 returned
+
+In this case, we'll need to pull the missing image ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
Review Comment:
```suggestion
In this case, we'll need to pull the missing image.For example, for Python 3.10 we need to use ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #35813:
URL: https://github.com/apache/airflow/pull/35813#discussion_r1403251382
##########
BREEZE.rst:
##########
@@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved
The similar issue could happen if you are behind an HTTP/HTTPS proxy and your access to required websites are
blocked by it, or your proxy setting has not been done properly.
+"No such image" Error:
+------------------------
+
+When running ``breeze start-airflow`` or commands that needs a base image, the following output might be observed:
+
+.. code-block:: bash
+
+ [+] Creating 3/0
+ ✔ Container docker-compose-rabbitmq-1 Running 0.0s
+ ✔ Container docker-compose-redis-1 Running 0.0s
+ ✔ Container docker-compose-postgres-1 Running 0.0s
+ [+] Building 0.0s (0/0) docker:desktop-linux
+ Error response from daemon: No such image: ghcr.io/apache/airflow/main/ci/python3.10:latest
+ Error 1 returned
+
+In this case, we'll need to push the missing image ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
Review Comment:
```suggestion
In this case, we'll need to pull the missing image ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1824305316
Actually this is not the right solution. The problem is that image should be automatically build if it is missing, there should never be the need to pull it manually.
Likely this might be a side effect of some change where not having an image triggers this problem. It might be result of some recent refactors I've done, but user should never need to pull the image of Breeze manually, user should not even have to know the image name, it's merely an implementation detaill
Was there any special you've done (cleaning docker etc.? @Lee-W - can you come up withe a scenario where this could happen ? I might try to reproduce it, but if you more ore less know how you got there, it could help to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1824276280
`No such image` Error also could be resolve by run one of this command, both of them also support `--python` option
```console
breeze ci-image pull
```
or
```console
breeze ci-image build
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1826277261
I found the problem and fixed it as part of https://github.com/apache/airflow/pull/35830
The problem is that the check only checked if the image should be upgraded (not whether it was pulled in the first place). I have also found another problem that prevented the image to be automatically build correctly (version suffix was not set to dev0 and there were conflicting requirements). Closing in favour or my PR :).
Thanks for raising the problem :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1826477533
Wow, it's a huge PR. thanks for helping out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #35813:
URL: https://github.com/apache/airflow/pull/35813#discussion_r1403255497
##########
BREEZE.rst:
##########
@@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved
The similar issue could happen if you are behind an HTTP/HTTPS proxy and your access to required websites are
blocked by it, or your proxy setting has not been done properly.
+"No such image" Error:
+------------------------
+
+When running ``breeze start-airflow`` or commands that needs a base image, the following output might be observed:
+
+.. code-block:: bash
+
+ [+] Creating 3/0
+ ✔ Container docker-compose-rabbitmq-1 Running 0.0s
+ ✔ Container docker-compose-redis-1 Running 0.0s
+ ✔ Container docker-compose-postgres-1 Running 0.0s
+ [+] Building 0.0s (0/0) docker:desktop-linux
+ Error response from daemon: No such image: ghcr.io/apache/airflow/main/ci/python3.10:latest
+ Error 1 returned
+
+In this case, we'll need to pull the missing image ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
Review Comment:
In this case, we'll need to pull the missing image. For example, for Python 3.10 we need to use ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1824658121
Ok. I think (maybe you can check) just adding check_and_rebuild_image_if_needed() when the start command run should solve the problem. It I lis for sure done in shell command and default one. Getting to plane now but should be easy to find and fix :) somewhere in 'developer_commands'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1826987001
> Yep, I'm saying that just to praise the good work, not to request to split the PR. It just might take sometime for me to take a deeper look
I am splitting it anyway to make it easier to review in small chunks. The current one is https://github.com/apache/airflow/pull/35875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1880202964
@RNHTTR -> Post on slack the whole output of what you have done. (and whether it was fresh install.
Generally speaking you should always:
a) rebase to latest main first
b) if you get build error when you are running breeze and choosing to rebuid image (or in `breeze ci-image build`) and get conflicts, you should try wuth `--upgrade-to-newer-dependencies` and breeze should actually advice you to do it. You should have YELLOW warning and an exact command to run starting with "if you see conflict error" or "if you see strange stacktraces"
I recommend to follow a) and b) - and make sure to read what breeze advice you get.
If you follow it and still got error - please post me the output of the command you run after adding `--verbose` flag.
Unfirtunately seeing just a `piece` of the output and even not knowing what command generated it and whether you got and followed the advice printed,does not hep too much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1879255311
I just got the following error...
```
9.887 ERROR: Cannot install apache-airflow[devel-ci]==2.8.0.dev0 because these package versions have conflicting dependencies.
9.887
9.887 The conflict is caused by:
9.887 apache-airflow[devel-ci] 2.8.0.dev0 depends on WTForms<3.1.0
9.887 The user requested (constraint) wtforms==3.1.1
...
Error response from daemon: No such image: ghcr.io/apache/airflow/main/ci/python3.10:latest
```
Then I ran `docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest` before running `breeze start-airflow` again. `breeze start-airflow` still gave me that error, but I guess it was able to fall back to the image I just pulled.
Now I'm getting some yarn error :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #35813: add guideline for missing image when running breeze
URL: https://github.com/apache/airflow/pull/35813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1826800001
Actually, that fix was separated out to a slightly smaller one https://github.com/apache/airflow/pull/35862
I **could** split it out even further if reviewers think it makes sense (I thin that would inflate the number of PRs artifficiall but if others think the PR is still too big to review, I am happy to split it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1825425883
Sounds good. Let me check where i can add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1826959897
Yep, I'm saying that just to praise the good work, not to request to split the PR. It just might take sometime for me to take a deeper look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #35813:
URL: https://github.com/apache/airflow/pull/35813#discussion_r1403256063
##########
BREEZE.rst:
##########
@@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved
The similar issue could happen if you are behind an HTTP/HTTPS proxy and your access to required websites are
blocked by it, or your proxy setting has not been done properly.
+"No such image" Error:
+------------------------
+
+When running ``breeze start-airflow`` or commands that needs a base image, the following output might be observed:
+
+.. code-block:: bash
+
+ [+] Creating 3/0
+ ✔ Container docker-compose-rabbitmq-1 Running 0.0s
+ ✔ Container docker-compose-redis-1 Running 0.0s
+ ✔ Container docker-compose-postgres-1 Running 0.0s
+ [+] Building 0.0s (0/0) docker:desktop-linux
+ Error response from daemon: No such image: ghcr.io/apache/airflow/main/ci/python3.10:latest
+ Error 1 returned
+
+In this case, we'll need to pull the missing image ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
Review Comment:
```suggestion
In this case, we'll need to pull the missing image. For example, for Python 3.10 we need to use ``ghcr.io/apache/airflow/main/ci/python3.10:latest`` as stated in the error message through ``docker pull ghcr.io/apache/airflow/main/ci/python3.10:latest``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add guideline for missing image when running breeze [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #35813:
URL: https://github.com/apache/airflow/pull/35813#issuecomment-1824629892
> Was there any special you've done (cleaning docker etc.? @Lee-W - can you come up withe a scenario where this could happen ? I might try to reproduce it, but if you more ore less know how you got there, it could help to fix it.
Yes, `docker system prune -a` or simply run `docker rmi ghcr.io/apache/airflow/main/ci/python3.10` and then `breeze start-airflow` will encounter such error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org