You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/02 20:28:25 UTC
[GitHub] [airflow] Osamaelgendy opened a new issue, #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Osamaelgendy opened a new issue, #27475:
URL: https://github.com/apache/airflow/issues/27475
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
I used puckel/docker-airflow latest image to build airflow web server in docker-compose file with other services.
The problem is when it starts the dagfile.py, this error pops up
I read that **Importing operators, sensors, hooks added in plugins via airflow.{operators,sensors,hooks}.<plugin_name> is no longer supported, and these extensions should just be imported as regular python modules**
but I don't know how to do this exactly, I already installed some python packages in the entrypoint.sh as it's written in the original repo
but how to install this?
### What you think should happen instead
I think it should be installed like any other python packages in the requirement.txt file, but i don't know how
### How to reproduce
_No response_
### Operating System
Windows
### Versions of Apache Airflow Providers
puckel/docker-airflow:latest
### Deployment
Docker-Compose
### Deployment details
**Docker-compose file**
version: "3.7"
services:
database:
image: mysql:8.0
platform: linux/amd64
extra_hosts:
- "somehost:162.242.195.82"
command:
- "--default-authentication-plugin=mysql_native_password"
environment:
- MYSQL_RANDOM_ROOT_PASSWORD=yes
- MYSQL_DATABASE=codetest
- MYSQL_USER=codetest
- MYSQL_PASSWORD=swordfish
- MYSQL_TIMEOUT = 300000
ports:
- "3306:3306"
restart: always
clickhouse:
image: "yandex/clickhouse-server"
environment:
- CLICKHOUSE_USER=codetest
- CLICKHOUSE_PASSWORD=swordfish
- CLICKHOUSE_DATABASE=codetest
ulimits:
nofile:
soft: 262144
hard: 262144
ports:
- "9999:9000" # native interface
- "8888:8123" # http interface
postgres:
image: postgres:9.6
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
logging:
options:
max-size: 10m
max-file: "3"
webserver:
image: puckel/docker-airflow:latest
#restart: always
depends_on:
- postgres
environment:
- INSTALL_MYSQL=y
- LOAD_EX=n
- EXECUTOR=Local
logging:
options:
max-size: 40m
max-file: "5"
volumes:
- ./dags:/usr/local/airflow/dags
- ./data:/usr/local/airflow/data
- ./logs:/usr/local/airflow/logs
- ./plugins:/usr/local/airflow/plugins
- ./requirements.txt:/usr/local/airflow/requirements.txt
- ./config/airflow.cfg:/opt/airflow/airflow.cfg
- ./docker-entrypoint.sh:/usr/local/airflow/docker-entrypoint.sh
entrypoint:
./docker-entrypoint.sh
ports:
- "8080:8080"
command: webserver
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1301496230
>
Thank you, I'll try another one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Corea commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
Corea commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1301343856
`puckel/docker-airflow` seems to be unmanaged for a long time. As it uses Airflow 1, please use different container provides Airflow 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1302724840
Now it gives me Permission denied: '/opt/airflow/logs/scheduler/2022-11-03 with the new container, I try to #sudo some commands to give permissions to 'airflow' user, but it won't accept 'airflow' as password I don't know why
I write this in entrypoint.sh **echo airflow | sudo -S visudo | ALL=(ALL) NOPASSWD: ALL**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1301189399
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1304956771
You had more issues than that - for example Python 3.6 is not supported in Airflow 2 and your line here tells that you tried to use it:
> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1304956248
> Just use official Airflow image https://airflow.apache.org/docs/docker-stack/index.html . Also read the docs. If you start from what you used in Puckel - scratch it completely and follow the examples found here https://airflow.apache.org/docs/docker-stack/build.html. DON'T try to copy what you've done with Puckel. You will have many problems like this. Try to understand what it does and rather than blindly copying instructions from your Puckel image without understanding it, understand what yiou want to achieve and achieve the same by following the examples and learning how the new image works.
>
> The way permissions were defined in Puckel were abomination. They were primitive, unsecure and not following modern recommendations and best practices of docker images. Scratch that. It's basically "garbage-in" -> "garbage out" approach and other things will not work if you attempt to do so.
>
> In your above case - airflow has no "sudo" capabilities by design. Read about the permissions in the documentation above - everything that airflow wants to use has to be "airflow" group writeable (this is to make the image open-shift compatible). Follow examples to switch the user using `USER` command in the Dockerfile. Make sure that all folders you mount to dockerfile are owned by "airflow" group if you want to make them writeable.
>
> Just follow the docs.. And DON't try to copy lines from Puckel.
Yes thank you for that, copying was a disaster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1304955746
I managed to solve these 2 issues, but it won't work, I spent a lot of time trying to configure this issue:
**File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "dag" does not exist
LINE 2: FROM dag
^**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1302735346
> `puckel/docker-airflow` seems to be unmanaged for a long time. As it uses Airflow 1, please use different container provides Airflow 2.
Now it gives me Permission denied: '/opt/airflow/logs/scheduler/2022-11-03 with the new container, I try to #sudo some commands to give permissions to 'airflow' user, but it won't accept 'airflow' as password I don't know why
I write this in entrypoint.sh **echo airflow | sudo -S visudo | ALL=(ALL) NOPASSWD: ALL**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1303043604
Just use official Airflow image https://airflow.apache.org/docs/docker-stack/index.html . Also read the docs. If you start from what you used in Puckel - scratch it completely and follow the examples found here https://airflow.apache.org/docs/docker-stack/build.html. DON'T try to copy what you've done with Puckel. You will have many problems like this. Try to understand what it does and rather than blindly copying instructions from your Puckel image without understanding it, understand what yiou want to achieve and achieve the same by following the examples and learning how the new image works.
The way permissions were defined in Puckel were abominations. They were primitive, unsecure and not following modern recommendations and best practices of docker images. Scratch that. It's basically "garbage-in" -> "garbage out" approache and other things will not work if you attempt to do so.
In your above case - airflow has no "sudo" capabilities by design. Read about the permissions in the documentation above - everything that airflow wants to use has to be "airflow" group writeable (this is to make the image open-shift compatible).
Follow examples to switch the user using `USER` command in the Dockerfile. Make sure that all folders you mount to dockerfile are owned by "airflow" group if you want to make them writeable.
Just follow the docs.. And DON't try to copy lines from Puckel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
URL: https://github.com/apache/airflow/issues/27475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1305009685
I'll check that, turned out the undefined table is a problem of metadata loading into the postgres db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org