You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/27 09:22:35 UTC
[GitHub] [airflow] yzliao-zip opened a new issue #21723: breeze shell fails with production image
yzliao-zip opened a new issue #21723:
URL: https://github.com/apache/airflow/issues/21723
### Apache Airflow version
2.2.3 (latest released)
### What happened
When use the production image for breeze shell, the container could not start. The error is `./breeze: line 3407: /Users/xxx/repos/github/airflow/.build/: Is a directory`
Files in `.build`:
```bash
❯ ls .build
dc_ci tmp_mssql_volume
```
### What you expected to happen
The container could start.
### How to reproduce
On the main branch (sha: 4e959358ac4ef), run `./breeze shell -I`.
Log:
```bash
❯ ./breeze shell -I
Good version of docker 20.10.12.
*************** PRODUCTION IMAGE *************************
@&&&&&&@
@&&&&&&&&&&&@
&&&&&&&&&&&&&&&&
&&&&&&&&&&
&&&&&&&
&&&&&&&
@@@@@@@@@@@@@@@@ &&&&&&
@&&&&&&&&&&&&&&&&&&&&&&&&&&
&&&&&&&&&&&&&&&&&&&&&&&&&&&&
&&&&&&&&&&&&
&&&&&&&&&
&&&&&&&&&&&&
@@&&&&&&&&&&&&&&&@
@&&&&&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
&&&&&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
&&&&&&
&&&&&&&
@&&&&&&&&
@&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
@&&&@ && @&&&&&&&&&&& &&&&&&&&&&&& && &&&&&&&&&& &&& &&& &&&
&&& &&& && @&& &&& && && &&& &&&@ &&& &&&&& &&&
&&& &&& && @&&&&&&&&&&&& &&&&&&&&&&& && && &&& &&& &&& &&@ &&&
&&&&&&&&&&& && @&&&&&&&&& && && &&@ &&& &&@&& &&@&&
&&& &&& && @&& &&&@ && &&&&&&&&&&& &&&&&&&&&&&& &&&& &&&&
&&&&&&&&&&&& &&&&&&&&&&&& &&&&&&&&&&&@ &&&&&&&&&&&& &&&&&&&&&&& &&&&&&&&&&&
&&& &&& && &&& && &&& &&&& &&
&&&&&&&&&&&&@ &&&&&&&&&&&& &&&&&&&&&&& &&&&&&&&&&& &&&& &&&&&&&&&&
&&& && && &&&& && &&& &&&& &&
&&&&&&&&&&&&& && &&&&@ &&&&&&&&&&&@ &&&&&&&&&&&& @&&&&&&&&&&& &&&&&&&&&&&
Unable to find image 'ghcr.io/apache/airflow/main/prod/python3.7:latest' locally
latest: Pulling from apache/airflow/main/prod/python3.7
6552179c3509: Already exists
c4887dad22fd: Already exists
83fecc69c367: Already exists
a9d24969a226: Already exists
af77bcde342c: Already exists
3d8819813184: Pulling fs layer
42a0553bd411: Pulling fs layer
a39d5bf6206f: Pulling fs layer
64f982711403: Pulling fs layer
c5ddb3493dbf: Pulling fs layer
ad7d5ec7545b: Pulling fs layer
2ea32c3d676b: Pulling fs layer
56a6fcee9116: Pulling fs layer
64f982711403: Waiting
ad7d5ec7545b: Waiting
2ea32c3d676b: Waiting
c5ddb3493dbf: Waiting
42a0553bd411: Verifying Checksum
42a0553bd411: Download complete
3d8819813184: Verifying Checksum
3d8819813184: Download complete
c5ddb3493dbf: Verifying Checksum
c5ddb3493dbf: Download complete
ad7d5ec7545b: Verifying Checksum
ad7d5ec7545b: Download complete
2ea32c3d676b: Verifying Checksum
2ea32c3d676b: Download complete
a39d5bf6206f: Verifying Checksum
a39d5bf6206f: Download complete
56a6fcee9116: Verifying Checksum
56a6fcee9116: Download complete
3d8819813184: Pull complete
42a0553bd411: Pull complete
64f982711403: Verifying Checksum
64f982711403: Download complete
a39d5bf6206f: Pull complete
64f982711403: Pull complete
c5ddb3493dbf: Pull complete
ad7d5ec7545b: Pull complete
2ea32c3d676b: Pull complete
56a6fcee9116: Pull complete
Digest: sha256:e569a99553e07d79f3f9d5cf876c3ca188ef0ee04ee3e1a75ed81248b2de3df4
Status: Downloaded newer image for ghcr.io/apache/airflow/main/prod/python3.7:latest
Use production image.
Branch name: main
Docker image: ghcr.io/apache/airflow/main/prod/python3.7
Airflow source version: 2.3.0.dev0
Python version: 3.7
Backend: sqlite
<skip cheatsheet messages>
Unable to find image 'ghcr.io/apache/airflow/main/ci/python3.7:latest' locally
latest: Pulling from apache/airflow/main/ci/python3.7
6552179c3509: Already exists
c4887dad22fd: Already exists
83fecc69c367: Already exists
a9d24969a226: Already exists
af77bcde342c: Already exists
b8797f7aaf3b: Pull complete
7aa60a7eb179: Pull complete
e17aa65b6f9e: Pull complete
cfeac562c98a: Pull complete
20974282c45c: Pull complete
e8a77142806e: Pull complete
c88283731f11: Pull complete
f264c85f13c8: Pull complete
8b94053432f4: Pull complete
560f0ba4744f: Pull complete
fe471bf11bf5: Pull complete
41c8eee50012: Pull complete
219c001c0d84: Pull complete
bd9156002c1b: Pull complete
d0a64492c53e: Pull complete
ad72a746ad5d: Pull complete
f62bc8725185: Pull complete
f67fe8db413d: Pull complete
bb264f6b4dc0: Pull complete
c16ed58d1eb1: Pull complete
d957f17ad7c5: Pull complete
d05f4914d743: Pull complete
4fbd9556d3fc: Pull complete
25b10cf6bcba: Pull complete
935a9d7ab47f: Pull complete
970ddab8a3c2: Pull complete
76f00f4a2fc3: Pull complete
9650de94a70c: Pull complete
608317699c60: Pull complete
Digest: sha256:be571982eb005551c806108047b1780a3229379cc8e2dae160207c3bf0e69b31
Status: Downloaded newer image for ghcr.io/apache/airflow/main/ci/python3.7:latest
Checking resources.
* Memory available 11GB. OK.
* Cpus available 8. OK.
* Disk available 44GB. OK.
Resource check successful.
./breeze: line 3407: /Users/xxxx/repos/github/airflow/.build/: Is a directory
ERROR: The previous step completed with error. Please take a look at output above
```
### Operating System
Mac OS 12.1
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21723: breeze shell fails with production image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21723:
URL: https://github.com/apache/airflow/issues/21723#issuecomment-1053528769
> Thank you for the clarification! My use case is using breeze to test the dag locally. Our production environment is on k8s and uses the production image. So I was trying to reproduce the same environment locally for test. It will be convenient to load production docker image via breeze, with the file system mounted, for local development.
This is not really convenient/easy and you will not get what you want, because prod image of Airlfow has different way of installing sources. Sources in CI are just mounted in /opt/airflow. But Production image "airflow" is installed from packages (separately airflow packages amd separately provider package) - so mounting sources to prod image is not a good idea, because it contains both providers, and airlfow sources from main - where in the production image sources of airlfow are diffferent than sources of providers (they might come from completely different commits)
I guess you do not want to mount "airlfow" sources, but your own sources, and in order to do that it is FAR easier to use plain docker commands rather than breeze:
```
docker run -it -v <your_sources>:/your_sources apache/airflow:2.2.4-python3.7 bash
```
^^ this will mount your sources to `/your_sources` in the image. No need to get Breeze involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21723: breeze shell fails with production image
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21723:
URL: https://github.com/apache/airflow/issues/21723
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yzliao-zip commented on issue #21723: breeze shell fails with production image
Posted by GitBox <gi...@apache.org>.
yzliao-zip commented on issue #21723:
URL: https://github.com/apache/airflow/issues/21723#issuecomment-1053141683
Thank you for the reply @potiuk . I rebase the main branch to the latest commit and still see the same error by running `./breeze shell -I`. Do you recommend to not use `-I` flag for the shell command? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21723: breeze shell fails with production image
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21723:
URL: https://github.com/apache/airflow/issues/21723
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21723: breeze shell failed with production image
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21723:
URL: https://github.com/apache/airflow/issues/21723#issuecomment-1047489617
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yzliao-zip commented on issue #21723: breeze shell fails with production image
Posted by GitBox <gi...@apache.org>.
yzliao-zip commented on issue #21723:
URL: https://github.com/apache/airflow/issues/21723#issuecomment-1053439708
Thank you for the clarification! My use case is using breeze to test the dag locally. Our production environment is on k8s and uses the production image. So I was trying to reproduce the same environment locally for test. It will be convenient to load production docker image via breeze, with the file system mounted, for local development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21723: breeze shell fails with production image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21723:
URL: https://github.com/apache/airflow/issues/21723#issuecomment-1053413716
Added better handling of this case in #21848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21723: breeze shell fails with production image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21723:
URL: https://github.com/apache/airflow/issues/21723#issuecomment-1052766285
Breeze is a development tool and it only works for latest "main` version reliably. Please check-out the `main` version. It will always install with the latest sources not with particular version.
You can also add `--use-airflow-version 2.2.3` to get particular version installed when you enter breeze, but it also might break occasionally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21723: breeze shell fails with production image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21723:
URL: https://github.com/apache/airflow/issues/21723#issuecomment-1053401841
There is no need to that and I will remove it now. It's a bit of a "useless" combination.
If you want, you can simply enter prod image using the regular docker run command as described in https://airflow.apache.org/docs/docker-stack/entrypoint.html#executing-commands
For example:
```
docker run -it apache/airflow:2.2.4-python3.7 bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org