You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hrvoje-intuita (via GitHub)" <gi...@apache.org> on 2023/02/03 11:20:58 UTC
[GitHub] [airflow] hrvoje-intuita opened a new issue, #29351: Apache Breeze - Python packages problems.
hrvoje-intuita opened a new issue, #29351:
URL: https://github.com/apache/airflow/issues/29351
### Apache Airflow version
main (development)
### What happened
Hi,
I'm using Apache Breeze on WSL2 Windows 11 machine.
Breeze is installed without problems and runs images fine. When running with empty init.sh all runs well and Airflow starts ok for this command:
```
breeze start-airflow --python 3.10 --load-example-dags --backend postgres --postgres-version 14 --use-airflow-version 2.5.0 │
--airflow-constraints-reference "https://raw.githubusercontent.com/apache/airflow/constraints-2.5.0/constraints-3.10.txt" |
--airflow-extras [pandas,amazon,postgres,sftp]
```
I'm trying to install Python packages used in DAGs with file
`files/airflow-breeze-config/init.sh`
as advised [here](https://github.com/apache/airflow/blob/08ec820b167ab3b2489ad8e4fb900d9390cfea49/BREEZE.rst#customizing-your-environment).
My init.sh:
```
pip install --upgrade pip
pip install -r /opt/airflow/dags/requirements.txt
```
First command runs fine, second one gives error:
> Collecting airflow==0.6
> Downloading airflow-0.6.tar.gz (1.2 kB)
> Preparing metadata (setup.py) ... error
> error: subprocess-exited-with-error
>
> × python setup.py egg_info did not run successfully.
> │ exit code: 1
> ╰─> [14 lines of output]
> running egg_info
> creating /tmp/pip-pip-egg-info-6vzwl974/airflow.egg-info
> writing /tmp/pip-pip-egg-info-6vzwl974/airflow.egg-info/PKG-INFO
> writing dependency_links to /tmp/pip-pip-egg-info-6vzwl974/airflow.egg-info/dependency_links.txt
> writing top-level names to /tmp/pip-pip-egg-info-6vzwl974/airflow.egg-info/top_level.txt
My requirements.txt looks like this:
```
pyOpenSSL==22.0.0
boto3==1.26.50
botocore==1.23.34
jsonpath_ng==1.5.3
numpy==1.24.1
pandas==1.5.2
pendulum==2.1.2
psycopg2==2.9.5
psycopg2_binary==2.9.5
python_dateutil==2.8.2
redshift_connector==2.0.909
requests==2.25.1
```
So I don't know why is trying to install `airflow-0.6.tar.gz` when it's not in list of packages in my requirements file?
I've tried doing `breez stop` and `breez cleanup` which didn't help.
What would be best way to install all Python packages dependencies for all DAGs anyway?
Thanks,
H
### What you think should happen instead
It should install Python packages and run Airflow.
### How to reproduce
In `init.sh` put:
```
pip install --upgrade pip
pip install -r /opt/airflow/dags/requirements.txt
```
Create `requirements.txt` and put it in `dags/requirements.txt`. Add list of some Python packages that needs to be installed:
```
pyOpenSSL==22.0.0
boto3==1.26.50
botocore==1.23.34
jsonpath_ng==1.5.3
numpy==1.24.1
pandas==1.5.2
pendulum==2.1.2
psycopg2==2.9.5
psycopg2_binary==2.9.5
python_dateutil==2.8.2
redshift_connector==2.0.909
requests==2.25.1
```
You can now run:
```
breeze start-airflow --python 3.10 --load-example-dags --backend postgres --postgres-version 14 --use-airflow-version 2.5.0 │
--airflow-constraints-reference "https://raw.githubusercontent.com/apache/airflow/constraints-2.5.0/constraints-3.10.txt" |
--airflow-extras [pandas,amazon,postgres,sftp]
```
### Operating System
Ubuntu 22.04 on WSL2 Windows 11
### Versions of Apache Airflow Providers
pandas,amazon,postgres,sftp
### Deployment
Docker-Compose
### Deployment details
Good version of Docker: 20.10.23.
Good version of docker-compose: 2.15.1
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29351: Apache Breeze - Problem installing Python packages with init.sh script
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29351: Apache Breeze - Problem installing Python packages with init.sh script
URL: https://github.com/apache/airflow/issues/29351
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29351: Apache Breeze - Python packages problems.
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29351:
URL: https://github.com/apache/airflow/issues/29351#issuecomment-1415733793
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org