You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/16 21:57:28 UTC
[GitHub] [airflow] frenchwr opened a new issue #17641: Import Errors Flapping
frenchwr opened a new issue #17641:
URL: https://github.com/apache/airflow/issues/17641
<!--
Welcome to Apache Airflow!
Please complete the next sections or the issue will be closed.
-->
**Apache Airflow version**: 2.1.2
<!-- AIRFLOW VERSION IS MANDATORY -->
**OS**: Amazon Linux release 2 (Karoo)
<!-- MANDATORY! You can get it via `cat /etc/oss-release` for example -->
**Apache Airflow Provider versions**:
```
apache-airflow-providers-amazon==2.0.0
apache-airflow-providers-apache-hdfs==2.0.0
apache-airflow-providers-apache-hive==2.0.0
apache-airflow-providers-celery==2.0.0
apache-airflow-providers-docker==2.0.0
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-google==4.0.0
apache-airflow-providers-hashicorp==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-microsoft-mssql==2.0.0
apache-airflow-providers-odbc==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-sftp==2.0.0
apache-airflow-providers-slack==4.0.0
apache-airflow-providers-sqlite==2.0.0
apache-airflow-providers-ssh==2.0.0
```
<!-- You can use `pip freeze | grep apache-airflow-providers` (you can leave only relevant ones)-->
**Deployment**:
Running on single AWS EC2 instance (connected to an AWS EMR cluster) with [Supervisor](http://supervisord.org/running.html) used to manage airflow processes. We use CeleryWorker and postgresql for our backend db.
<!-- e.g. Virtualenv / VM / Docker-compose / K8S / Helm Chart / Managed Airflow Service -->
<!-- Please include your deployment tools and versions: docker-compose, k8s, helm, etc -->
**What happened**:
A handful of parsing errors are going in and out of the `import_error` table. I uncovered this while specifically trying to address a parsing error from botocore: `botocore.exceptions.ProfileNotFound: The config profile (nonprod) could not be found`. The stacktrace from this error was originally in both the `import_error` table and also appeared in the scheduler logs for the DAG producing this error.
After addressing this error (Supervisor needed additional environment vars to be set so it knew where to find the AWS creds file), the error disappeared from the scheduler logs (both for the individual DAG logs and in the report generated by the DAG File Processor). However, the error flaps in and out of the `import_error` table. I notice that immediately after the DAG processing loop completes (we have it set to run every two minutes) the error disappears from the `import_error` table for about 5-15 seconds before returning again, and remaining there until the next DAG processing loop.
<!-- Please include exact error messages if you can -->
**What you expected to happen**:
I expect the parsing error to disappear permanently from the `import_error` table.
<!-- What do you think went wrong? -->
**How to reproduce it**:
I have not attempted to create a minimal reproducer.
<!--
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images/screen-casts etc. by drag-dropping the image here.
-->
**Anything else we need to know**:
It's unclear to me whether this is a bug or there is something in our Supervisor config that needs to be further tweaked, so I'm reaching out to see if anyone has a hunch of what might be occurring (e.g. are the previous import errors perhaps getting cached some place and then re-read? or maybe the old errors are stored some place in the serialized version of the DAG?). I appreciate any help or tips!
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here inside fenced
``` ``` blocks or inside a foldable details tag if it's long:
<details><summary>x.log</summary> lots of stuff </details>
-->
**Are you willing to submit a PR?**
Sure!
<!---
This is absolutely not required, but we are happy to guide you in contribution process
especially if you already have a good understanding of how to implement the fix.
Airflow is a community-managed project and we love to bring new contributors in.
Find us in #airflow-how-to-pr on Slack!
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17641: Import Errors Flapping
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17641:
URL: https://github.com/apache/airflow/issues/17641#issuecomment-902163270
It almost looks like you have two schedulers running at the same time - one with the properly set environment variable and the other without. Are you sure you do not have two schedulers running?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17641: Import Errors Flapping
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17641:
URL: https://github.com/apache/airflow/issues/17641#issuecomment-899844149
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17641: Import Errors Flapping
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17641:
URL: https://github.com/apache/airflow/issues/17641#issuecomment-902961426
Glad it helped :).
> “How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth?”
Sherlock Holmes Quote
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] frenchwr closed issue #17641: Import Errors Flapping
Posted by GitBox <gi...@apache.org>.
frenchwr closed issue #17641:
URL: https://github.com/apache/airflow/issues/17641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] frenchwr commented on issue #17641: Import Errors Flapping
Posted by GitBox <gi...@apache.org>.
frenchwr commented on issue #17641:
URL: https://github.com/apache/airflow/issues/17641#issuecomment-902829118
> It almost looks like you have two schedulers running at the same time - one with the properly set environment variable and the other without. Are you sure you do not have two schedulers running?
I had checked this before _on the same instance_ and was getting ready to post a reply when I realized we had left an old dev EC2 instance running. That instance was using the same backend postgres db and was generating the errors. Stopping the instance resolved the issue.
Thanks for weighing in pointing me in the right direction!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org