You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/23 16:48:38 UTC
[GitHub] [airflow] baryluk opened a new pull request #17195: Be verbose about failure to import airflow_local_settings
baryluk opened a new pull request #17195:
URL: https://github.com/apache/airflow/pull/17195
Currently, if the module exists, but has errors (for example syntax
error, or transitive import of module that does not exist),
the airflow scheduler will not show any error.
In my opinion `ImportError` of `airflow_local_settings` should
be a fatal error, but let it be a warning for now.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-885771910
Yeah. exactly - failure to import local_settings at all is OK. but if the import inside local_settings fails, this should be a fatal error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baryluk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
baryluk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886937047
@potiuk Thanks for your patience and quick review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-885770442
I also think it should be a fatal error. Anyone thinks otherwise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #17195:
URL: https://github.com/apache/airflow/pull/17195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baryluk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
baryluk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886678537
> Some static checks and tests are failing @baryluk
@potiuk should be fixed. can rerun CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baryluk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
baryluk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-885771223
Actually transitive imports might still be not handled correctly.
Let me address this by checking the ModuleNotFoundError.name field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-885788421
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886241074
Some static checks and tests are failing @baryluk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baryluk commented on a change in pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
baryluk commented on a change in pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#discussion_r675720601
##########
File path: airflow/settings.py
##########
@@ -431,8 +431,15 @@ def import_local_settings():
del globals()["policy"]
log.info("Loaded airflow_local_settings from %s .", airflow_local_settings.__file__)
- except ImportError:
- log.debug("Failed to import airflow_local_settings.", exc_info=True)
+ except ModuleNotFoundError as e:
+ if e.name == "airflow_local_settings":
+ log.debug("No airflow_local_settings to import.", exc_info=True)
+ else:
+ log.critical("Failed to import airflow_local_settings due to a transitive module not found error.", exc_info=True)
+ raise e
Review comment:
Thanks, that is indeed better. I was not aware of that feature of Python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#discussion_r675709293
##########
File path: airflow/settings.py
##########
@@ -432,7 +432,12 @@ def import_local_settings():
log.info("Loaded airflow_local_settings from %s .", airflow_local_settings.__file__)
except ImportError:
- log.debug("Failed to import airflow_local_settings.", exc_info=True)
+ log.warning("Failed to import airflow_local_settings.", exc_info=True)
+ except ModuleNotFoundError as e:
Review comment:
ModuleNotFoundError derirves from ImportError so it should be higher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baryluk commented on a change in pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
baryluk commented on a change in pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#discussion_r675711455
##########
File path: airflow/settings.py
##########
@@ -432,7 +432,12 @@ def import_local_settings():
log.info("Loaded airflow_local_settings from %s .", airflow_local_settings.__file__)
except ImportError:
- log.debug("Failed to import airflow_local_settings.", exc_info=True)
+ log.warning("Failed to import airflow_local_settings.", exc_info=True)
+ except ModuleNotFoundError as e:
Review comment:
Done. Also made it critical and re-raise the exception.
This is no different that other possible errors, like `SyntaxError`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886936380
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886222289
Closed/reopened to re-trigger the build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886817074
Still static check problems. I highly recommend installing pre-commit, it will fix those kind of problems for you automatically :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #17195:
URL: https://github.com/apache/airflow/pull/17195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886935895
seems like transient errors only. Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-885767543
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886741956
We had `main` broken so the test failures are unrelated. Those `main` errors were fixed this morning - can you please rebase to latest main ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-885771910
Yeah. exactly - failure to import local_settings at all is OK. but if the import inside local_settings fails, this should be a fatal error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-885770442
I also think it should be a hard error. Anyone thinks otherwise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#discussion_r675715236
##########
File path: airflow/settings.py
##########
@@ -431,8 +431,15 @@ def import_local_settings():
del globals()["policy"]
log.info("Loaded airflow_local_settings from %s .", airflow_local_settings.__file__)
- except ImportError:
- log.debug("Failed to import airflow_local_settings.", exc_info=True)
+ except ModuleNotFoundError as e:
+ if e.name == "airflow_local_settings":
+ log.debug("No airflow_local_settings to import.", exc_info=True)
+ else:
+ log.critical("Failed to import airflow_local_settings due to a transitive module not found error.", exc_info=True)
+ raise e
+ except ImportError as e:
+ log.critical("Failed to import airflow_local_settings.", exc_info=True)
+ raise e
Review comment:
```suggestion
raise
```
##########
File path: airflow/settings.py
##########
@@ -431,8 +431,15 @@ def import_local_settings():
del globals()["policy"]
log.info("Loaded airflow_local_settings from %s .", airflow_local_settings.__file__)
- except ImportError:
- log.debug("Failed to import airflow_local_settings.", exc_info=True)
+ except ModuleNotFoundError as e:
+ if e.name == "airflow_local_settings":
+ log.debug("No airflow_local_settings to import.", exc_info=True)
+ else:
+ log.critical("Failed to import airflow_local_settings due to a transitive module not found error.", exc_info=True)
+ raise e
Review comment:
```suggestion
raise
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baryluk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
baryluk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886789921
> We had `main` broken so the test failures are unrelated. Those `main` errors were fixed this morning - can you please rebase to latest main ?
Rebased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baryluk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
baryluk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-885789822
> Nice! Really useful to detect some hard-to-debug problems!
It did bite me of course, that is why I am suggesting this change. I replaced `from airflow.contrib.kubernetes.pod import Pod` with `from airflow.kubernetes.pod import Pod` blindly. And was surirpised that some spark kubernetes jobs could not find secrets that were provided in the `pod_mutation_hook` in the `airflow_local_settings.py`, with no logging of the issue at all. :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17195: Be verbose about failure to import airflow_local_settings
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17195:
URL: https://github.com/apache/airflow/pull/17195#issuecomment-886828609
And tests are still failing (this time settings tests, so you will need to fix it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org