You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/02 20:28:25 UTC

[GitHub] [airflow] Osamaelgendy opened a new issue, #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Osamaelgendy opened a new issue, #27475:
URL: https://github.com/apache/airflow/issues/27475

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   I used puckel/docker-airflow latest image to build airflow web server in docker-compose file with other services.
   The problem is when it starts the dagfile.py, this error pops up
   I read that **Importing operators, sensors, hooks added in plugins via airflow.{operators,sensors,hooks}.<plugin_name> is no longer supported, and these extensions should just be imported as regular python modules**
   
   but I don't know how to do this exactly, I already installed some python packages in the entrypoint.sh as it's written in the original repo
   
   but how to install this?
   
   ### What you think should happen instead
   
   I think it should be installed like any other python packages in the requirement.txt file, but i don't know how
   
   ### How to reproduce
   
   _No response_
   
   ### Operating System
   
   Windows
   
   ### Versions of Apache Airflow Providers
   
   puckel/docker-airflow:latest
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   **Docker-compose file**
   version: "3.7"
   
   services:
     database:
       image: mysql:8.0
       platform: linux/amd64
       extra_hosts:
         - "somehost:162.242.195.82"
       command:
         - "--default-authentication-plugin=mysql_native_password"
       environment:
         - MYSQL_RANDOM_ROOT_PASSWORD=yes
         - MYSQL_DATABASE=codetest
         - MYSQL_USER=codetest
         - MYSQL_PASSWORD=swordfish
         - MYSQL_TIMEOUT = 300000
       ports:
         - "3306:3306"
         
       restart: always
   
       
   
     clickhouse:
       image: "yandex/clickhouse-server"
       environment:
         - CLICKHOUSE_USER=codetest
         - CLICKHOUSE_PASSWORD=swordfish
         - CLICKHOUSE_DATABASE=codetest
       ulimits:
         nofile:
           soft: 262144
           hard: 262144
       ports:
         - "9999:9000" # native interface
         - "8888:8123" # http interface
   
     postgres:
           image: postgres:9.6
           environment:
               - POSTGRES_USER=airflow
               - POSTGRES_PASSWORD=airflow
               - POSTGRES_DB=airflow
           logging:
               options:
                   max-size: 10m
                   max-file: "3"
   
     webserver:
           image: puckel/docker-airflow:latest
           #restart: always
           depends_on:
               - postgres
           environment:
               - INSTALL_MYSQL=y
               - LOAD_EX=n
               - EXECUTOR=Local
           logging:
               options:
                   max-size: 40m
                   max-file: "5"
           volumes:
               - ./dags:/usr/local/airflow/dags
               - ./data:/usr/local/airflow/data
               - ./logs:/usr/local/airflow/logs
               - ./plugins:/usr/local/airflow/plugins
               - ./requirements.txt:/usr/local/airflow/requirements.txt
               - ./config/airflow.cfg:/opt/airflow/airflow.cfg
               - ./docker-entrypoint.sh:/usr/local/airflow/docker-entrypoint.sh
           entrypoint:
               ./docker-entrypoint.sh
               
           ports:
               - "8080:8080"
           command: webserver
           healthcheck:
               test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
               interval: 30s
               timeout: 30s
               retries: 3
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1301496230

   > 
   
   Thank you, I'll try another one 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Corea commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
Corea commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1301343856

   `puckel/docker-airflow` seems to be unmanaged for a long time. As it uses Airflow 1, please use different container provides Airflow 2. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1302724840

   Now it gives me  Permission denied: '/opt/airflow/logs/scheduler/2022-11-03 with the new container, I try to #sudo some commands to give permissions to 'airflow' user, but it won't accept 'airflow' as password I don't know why
   
   I write this in entrypoint.sh **echo airflow | sudo -S visudo | ALL=(ALL) NOPASSWD: ALL**


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1301189399

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1304956771

   You had more issues than that - for example Python 3.6 is not supported in Airflow 2 and your line here tells that you tried to use it:
   
   > File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1304956248

   > Just use official Airflow image https://airflow.apache.org/docs/docker-stack/index.html . Also read the docs. If you start from what you used in Puckel - scratch it completely and follow the examples found here https://airflow.apache.org/docs/docker-stack/build.html. DON'T try to copy what you've done with Puckel. You will have many problems like this. Try to understand what it does and rather than blindly copying instructions from your Puckel image without understanding it, understand what yiou want to achieve and achieve the same by following the examples and learning how the new image works.
   > 
   > The way permissions were defined in Puckel were abomination. They were primitive, unsecure and not following modern recommendations and best practices of docker images. Scratch that. It's basically "garbage-in" -> "garbage out" approach and other things will not work if you attempt to do so.
   > 
   > In your above case - airflow has no "sudo" capabilities by design. Read about the permissions in the documentation above - everything that airflow wants to use has to be "airflow" group writeable (this is to make the image open-shift compatible). Follow examples to switch the user using `USER` command in the Dockerfile. Make sure that all folders you mount to dockerfile are owned by "airflow" group if you want to make them writeable.
   > 
   > Just follow the docs.. And DON't try to copy lines from Puckel.
   
   Yes thank you for that, copying was a disaster 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1304955746

   I managed to solve these 2 issues, but it won't work, I spent a lot of time trying to configure this issue:
   
    **File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
       cursor.execute(statement, parameters)
   psycopg2.errors.UndefinedTable: relation "dag" does not exist
   LINE 2: FROM dag
                ^**


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1302735346

   > `puckel/docker-airflow` seems to be unmanaged for a long time. As it uses Airflow 1, please use different container provides Airflow 2.
   
   Now it gives me Permission denied: '/opt/airflow/logs/scheduler/2022-11-03 with the new container, I try to #sudo some commands to give permissions to 'airflow' user, but it won't accept 'airflow' as password I don't know why
   
   I write this in entrypoint.sh **echo airflow | sudo -S visudo | ALL=(ALL) NOPASSWD: ALL**
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1303043604

   Just use official Airflow image https://airflow.apache.org/docs/docker-stack/index.html . Also read the docs. If you start from what you used in Puckel - scratch it completely and follow the examples found here https://airflow.apache.org/docs/docker-stack/build.html. DON'T try to copy what you've done with Puckel. You will have many problems like this. Try to understand what it does and rather than blindly copying instructions from your Puckel image without understanding it, understand what yiou want to achieve and achieve the same by following the examples and learning how the new image works.
   
   The way permissions were defined in Puckel were abominations. They  were primitive, unsecure and not following modern recommendations and best practices of docker images. Scratch that. It's basically "garbage-in" -> "garbage out"  approache and other things will not work if you attempt to do so.
   
   In your above case - airflow has no "sudo" capabilities by design. Read about the permissions in the documentation above - everything that airflow wants to use has to be "airflow" group writeable (this is to make the image open-shift compatible). 
   Follow examples to switch the user using  `USER` command in the Dockerfile. Make sure that all folders you mount to dockerfile are owned by "airflow" group if you want to make them writeable.
   
   Just follow the docs.. And DON't try to copy lines from Puckel.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ephraimbuddy closed issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'
URL: https://github.com/apache/airflow/issues/27475


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Osamaelgendy commented on issue #27475: ModuleNotFoundError: No module named 'airflow.operators.PythonOperators'

Posted by GitBox <gi...@apache.org>.
Osamaelgendy commented on issue #27475:
URL: https://github.com/apache/airflow/issues/27475#issuecomment-1305009685

   I'll check that, turned out the undefined table is a problem of metadata loading into the postgres db


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org