You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/04 08:52:40 UTC
[GitHub] [airflow] JavierLopezT opened a new issue #15649: Dag Import error in UI with unreal message Airflow 2.02
JavierLopezT opened a new issue #15649:
URL: https://github.com/apache/airflow/issues/15649
I have just updated from Airflow 1.10.14 to Airflow 2.0.2 and I am getting an error of this kind in a lot of dags. I have several python scripts in my dags subfolders. Here is the error I get in the UI:
However, the code of the DAG is the following:
```
from utilities.files import get_arguments_python_script
dag_route, dag_name, ts = get_arguments_python_script()
dag_globals = Variable.get('variables_secret', deserialize_json=True)
```
I am using airflow with docker-compose and I have already restarted it several times and the error persists. Moreover, if I click on the DAG, I can successfully enter it and even run the script. Also, in the logs of the containers of the webserver and the scheduler, there is no error logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-849548067
Are you running the "get_pandas_df" at the top-level of your DAG file? if so, then yes - scheduler will try to execute it while parsing the DAG. DAG is "imported" and "executed" by the scheduler regularly. All the processing should happen in the operator's "execute()'' methods (either operators you use or the custom operators you write).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834174959
Also, looks like the issue was introduced in 2.0.2. Because migration back on 2.0.1 seems to solve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834185746
UPDATE: I see you pasted it above.
@codenamestif Looking at the fragment of the `s3_adyen.py` you pasted
Looking at the error description and the piece of code printed, it almost looks like the end of line after the python_script() was somehow stripped out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-849554693
> Are you running the "get_pandas_df" at the top-level of your DAG file? if so, then yes - scheduler will try to execute it while parsing the DAG. Python DAG file is "imported" and "executed" by the scheduler regularly. All the processing should happen in the operator's "execute()'' methods (either operators you use or the custom operators you write).
No, nothing is in the DAG file but the definition of the operators, as the best practices state. The `get_pandas_df` is in a python file that is executed by a custom BashOperator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-849562299
Is the code where you have `redshift =` in the `py_get_query.py` file? If so the scheduler is still going to parse all python files to look for dags, and parsing = running any and all code at the top level of the file.
You can either add the non-dag python files to the .airflowignore file https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html?highlight=airflowignore#loading-dags
Or better yet would be to not place non-DAG files in the dags folder and place them somewhere else in the Python include path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-849554693
> Are you running the "get_pandas_df" at the top-level of your DAG file? if so, then yes - scheduler will try to execute it while parsing the DAG. Python DAG file is "imported" and "executed" by the scheduler regularly. All the processing should happen in the operator's "execute()'' methods (either operators you use or the custom operators you write).
Thanks for answering Jarek. No, nothing is in the DAG file but the definition of the operators, as the best practices state. The `get_pandas_df` is in a python file that is executed by a custom BashOperator.
The files structure is the following:
```
dags
import_blog
dag_import_blog.py
py_get_query.py
```
In dag_import_blog.py:
```
with DAG(blabla)
task_1 = BashOperator(task_id='t1',
bash_command='python3 /opt/airflow/dags/import/blog/py_get_query.py)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-832662119
@uranusjr it's not about parsing the file for sure. I have python file with some constants. I've got the error after adding more constants there. The old ones were resolved with no problems, but new ones were not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
ashb closed issue #15649:
URL: https://github.com/apache/airflow/issues/15649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-840428645
I tried the files in a fresh Airflow installation and could not reproduce the error. So (assuming the files are verbatim, which I’m unsure of since Git performs some text processing by default), this is likely an issue with your local setup (probably database?) and not a bug in Airflow itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-831803410
Can you upload the DAG file (verbatim, not copy-and-pasted) somewhere? I’m guessing the file contains some weird characters and cannot be correctly parsed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834185746
UPDATE: I see you pasted it above.
@JavierLopezT Looking at the fragment of the `s3_adyen.py` you pasted it almost looks like the end of line after the python_script() was somehow stripped out. As @uranusjr suggested - can you upload the fie somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834210879
Additional question: Are you sure you migrated the `DB` before running new version of Airflow ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834188618
Maybe this is the usual Windows/Linux EOL problem CR/LF?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834201044
> Also, looks like the issue was introduced in 2.0.2. Because migration back on 2.0.1 seems to solve the issue.
I can confirm that with 2.0.1 I did not have the issue with the exact same file.
Also, I am getting other errors. For instance, I have a query in a python file and there is an error on the UI saying that:
`"table relation does not exist"`
And I don't think that there should be any SQL checking before executing a dag.
Furthermore, I had to change the way I passed parameters to my python files because with `sys.argv[1]` I obtained in the UI errors of kind IndexError (that I didn't have way back in 1.10.14: https://apache-airflow.slack.com/archives/C0146STM600/p1617278017105300
And I had an ImportError as well: https://apache-airflow.slack.com/archives/C0146STM600/p1617788878119200
I think that there is definitely something not working as intended in the new scheduler. I am using Airflow with the official Docker image, by the way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834214369
> > Also, I am getting other errors. For instance, I have a query in a python file and there is an error on the UI saying that:
>
> I am afraid we will need to get more details and verbatim file to be able to investigate. Do you have a possibility of running again 2.0.1 -> 2.0.2 without disrupting your current installation (for example by copying your database) and proividing some more details: verbatim file, more information about your deployment/database used etc.) + detailed list of stactraces/errors/log file errors you have.
>
> It looks like you have some specific environment (are your files edited/stored on windows maybe)?
>
> Otherwise it is simply very difficult to reproduce.
Thanks for your interest. My files are edited in Mac with Pycharm. I have just tested Airflow 2.0 versions with a local docker-compose in my mac, with Postgres and Local Executor, and the exact same files work with Airflow 1.10.14 perfectly.
I did migrate the DB. I changed the version of Postgres in the docker-compose from 11.5 to 13 and I had to do on my Mac
```
brew upgrade postgresql
brew postgresql-upgrade-database
```
After that, I entered the container and run the upgrade db, getting a successful message:
`docker exec -it airflow_scheduler_1 bash --> airflow db upgrade`
I had done the db upgrade three times:
1.10.14 -> 2.0.0
2.0.0 -> 2.0.1
2.01 -> 2.0.2
But in 2.0.0 and 2.0.2 I have not been free of errors, with the exact same files all the time. I will try to upload verbatim files. Thanks again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834208855
> Also, I am getting other errors. For instance, I have a query in a python file and there is an error on the UI saying that:
I am afraid we will need to get more details and verbatim file to be able to investigate. Do you have a possibility of running again 2.0.1 -> 2.0.2 without disrupting your current installation (for example by copying your database) and proividing some more details: verbatim file, more information about your deployment/database used etc.) + detailed list of stactraces/errors/log file errors you have.
It looks like you have some specific environment (are your files edited/stored on windows maybe)?
Otherwise it is simply very difficult to reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834185746
@codenamestif can you paste the fragment of the `s3_adyen.py` file that is reported as wrong?
Looking at the error description and the piece of code printed, it almost looks like the end of line after the python_script() was somehow stripped out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-849548067
Are you running the "get_pandas_df" at the top-level of your DAG file? if so, then yes - scheduler will try to execute it while parsing the DAG. Python DAG file is "imported" and "executed" by the scheduler regularly. All the processing should happen in the operator's "execute()'' methods (either operators you use or the custom operators you write).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834242093
You can find the files here @potiuk @uranusjr:
https://github.com/JavierLopezT/airflow_2.02_error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-849520348
Another error
The code is:
I don't think this error should happen. For me, it seems that the scheduler is trying to execute somehow the query before actually running it, but I don't think this import error is useful at all and I don't see how this could be related to my local deployment.
Any ideas? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15649:
URL: https://github.com/apache/airflow/issues/15649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-832656786
I confirm the issue. It looks like there is some kind of partial import. Especially when there are 2 files and one file has import on another one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-849554693
> Are you running the "get_pandas_df" at the top-level of your DAG file? if so, then yes - scheduler will try to execute it while parsing the DAG. Python DAG file is "imported" and "executed" by the scheduler regularly. All the processing should happen in the operator's "execute()'' methods (either operators you use or the custom operators you write).
No, nothing is in the DAG file but the definition of the operators, as the best practices state. The `get_pandas_df` is in a python file that is executed by a custom BashOperator.
The files structure is the following:
```
dags
import_blog
dag_import_blog.py
py_get_query.py
```
In dag_import_blog.py:
```
with DAG(blabla)
task_1 = BashOperator(task_id='t1',
bash_command='python3 /opt/airflow/dags/import/blog/py_get_query.py)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-849554693
> Are you running the "get_pandas_df" at the top-level of your DAG file? if so, then yes - scheduler will try to execute it while parsing the DAG. Python DAG file is "imported" and "executed" by the scheduler regularly. All the processing should happen in the operator's "execute()'' methods (either operators you use or the custom operators you write).
No, nothing is in the DAG file but the definition of the operators, as the best practices state. The `get_pandas_df` is in a python file that is executed by a custom BashOperator.
dags
import_blog
dag_import_blog.py
py_get_query.py
In dag_import_blog.py:
```
with DAG(blabla)
task_1 = BashOperator(task_id='t1',
bash_command='python3 /opt/airflow/dags/import/blog/py_get_query.py)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-834214369
> > Also, I am getting other errors. For instance, I have a query in a python file and there is an error on the UI saying that:
>
> I am afraid we will need to get more details and verbatim file to be able to investigate. Do you have a possibility of running again 2.0.1 -> 2.0.2 without disrupting your current installation (for example by copying your database) and proividing some more details: verbatim file, more information about your deployment/database used etc.) + detailed list of stactraces/errors/log file errors you have.
>
> It looks like you have some specific environment (are your files edited/stored on windows maybe)?
>
> Otherwise it is simply very difficult to reproduce.
Thanks for your interest. My files are edited in Mac with Pycharm. I have just tested Airflow 2.0 versions with a local docker-compose in my mac, with Postgres and Local Executor, and the exact same files work with Airflow 1.10.14 perfectly.
I did migrate the DB. I changed the version of Postgres in the docker-compose from 11.5 to 13 and I had to do on my Mac
```
brew upgrade postgresql
brew postgresql-upgrade-database
```
After that, I entered the container and run the upgrade db, getting a successful message:
`docker exec -it airflow_scheduler_1 bash --> airflow db upgrade`
I had done the db upgrade twice, but in 2.0.0 and 2.0.2 I have not been free of errors, with the exact same files all the time. I will try to upload verbatim files. Thanks again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-848886152
I have upgraded the Dockerfile to 2.1.0, made `docker system prune -af` and `airflow db reset` as well as `airflow db init` several times. I don't have anymore the error in the first post, but I do have one very similar:
And the DAG works perfectly fine:
![Uploading Captura de pantalla 2021-05-26 a las 17.38.24.png…]()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on issue #15649: Dag Import error in UI with unreal message Airflow 2.02
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on issue #15649:
URL: https://github.com/apache/airflow/issues/15649#issuecomment-848886152
I have upgraded the Dockerfile to 2.1.0, made `docker system prune -af` and `airflow db reset` as well as `airflow db init` several times. I don't have anymore the error in the first post, but I do have one very similar:
And the DAG works perfectly fine:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org