You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/01 09:31:58 UTC
[GitHub] [airflow] jedcunningham commented on a change in pull request #17924: Display ImportErrors from all readable DAGs
jedcunningham commented on a change in pull request #17924:
URL: https://github.com/apache/airflow/pull/17924#discussion_r699499319
##########
File path: airflow/www/views.py
##########
@@ -678,18 +678,19 @@ def index(self):
for name, in dagtags
]
- import_errors = session.query(errors.ImportError).order_by(errors.ImportError.id).all()
+ import_errors = session.query(errors.ImportError).order_by(errors.ImportError.id)
- if import_errors:
- dag_filenames = {dag.fileloc for dag in dags}
- all_dags_readable = (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG) in user_permissions
+ if (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG) not in user_permissions:
+ # if the user doesn't have access to all DAGs, only display errors from visible DAGs
+ import_errors = import_errors.join(
+ DagModel, DagModel.fileloc == errors.ImportError.filename
+ ).filter(DagModel.dag_id.in_(filter_dag_ids))
- for import_error in import_errors:
- if all_dags_readable or import_error.filename in dag_filenames:
- flash(
- "Broken DAG: [{ie.filename}] {ie.stacktrace}".format(ie=import_error),
- "dag_import_error",
- )
+ for import_error in import_errors:
+ flash(
+ "Broken DAG: [{ie.filename}] {ie.stacktrace}".format(ie=import_error),
+ "dag_import_error",
+ )
Review comment:
```suggestion
for import_error in import_errors:
flash(
"Broken DAG: [{ie.filename}] {ie.stacktrace}".format(ie=import_error),
"dag_import_error",
)
```
I'm thinking we now want to be in the create_session contextmanager now that we are lazily iterating?
##########
File path: airflow/www/views.py
##########
@@ -678,18 +678,19 @@ def index(self):
for name, in dagtags
]
- import_errors = session.query(errors.ImportError).order_by(errors.ImportError.id).all()
+ import_errors = session.query(errors.ImportError).order_by(errors.ImportError.id)
- if import_errors:
- dag_filenames = {dag.fileloc for dag in dags}
- all_dags_readable = (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG) in user_permissions
+ if (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG) not in user_permissions:
+ # if the user doesn't have access to all DAGs, only display errors from visible DAGs
+ import_errors = import_errors.join(
+ DagModel, DagModel.fileloc == errors.ImportError.filename
+ ).filter(DagModel.dag_id.in_(filter_dag_ids))
- for import_error in import_errors:
- if all_dags_readable or import_error.filename in dag_filenames:
- flash(
- "Broken DAG: [{ie.filename}] {ie.stacktrace}".format(ie=import_error),
- "dag_import_error",
- )
+ for import_error in import_errors:
+ flash(
+ "Broken DAG: [{ie.filename}] {ie.stacktrace}".format(ie=import_error),
+ "dag_import_error",
+ )
Review comment:
Basically, `create_session` ends up calling `commit` then `close` (see [create_session source](https://github.com/apache/airflow/blob/eebfeec4fa798b728141dfdeb4cf70970c71067f/airflow/utils/session.py#L27-L37) but `close` should more or less be considered `reset` (see [sqlalchemy docs](https://docs.sqlalchemy.org/en/14/orm/session_basics.html#closing):
> When the Session is closed, it is essentially in the original state as when it was first constructed, and may be used again.
So by using the session again outside the context manager, nothing is doing "proper" cleanup on it at that point. Not a huge deal in this specific case, but 🤷♂️.
##########
File path: airflow/www/views.py
##########
@@ -678,18 +678,19 @@ def index(self):
for name, in dagtags
]
- import_errors = session.query(errors.ImportError).order_by(errors.ImportError.id).all()
+ import_errors = session.query(errors.ImportError).order_by(errors.ImportError.id)
- if import_errors:
- dag_filenames = {dag.fileloc for dag in dags}
- all_dags_readable = (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG) in user_permissions
+ if (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG) not in user_permissions:
+ # if the user doesn't have access to all DAGs, only display errors from visible DAGs
+ import_errors = import_errors.join(
+ DagModel, DagModel.fileloc == errors.ImportError.filename
+ ).filter(DagModel.dag_id.in_(filter_dag_ids))
- for import_error in import_errors:
- if all_dags_readable or import_error.filename in dag_filenames:
- flash(
- "Broken DAG: [{ie.filename}] {ie.stacktrace}".format(ie=import_error),
- "dag_import_error",
- )
+ for import_error in import_errors:
+ flash(
+ "Broken DAG: [{ie.filename}] {ie.stacktrace}".format(ie=import_error),
+ "dag_import_error",
+ )
Review comment:
Basically, `create_session` ends up calling `commit` then `close` (see [create_session source](https://github.com/apache/airflow/blob/eebfeec4fa798b728141dfdeb4cf70970c71067f/airflow/utils/session.py#L27-L37)) but `close` should more or less be considered `reset` (see [sqlalchemy docs](https://docs.sqlalchemy.org/en/14/orm/session_basics.html#closing):
> When the Session is closed, it is essentially in the original state as when it was first constructed, and may be used again.
So by using the session again outside the context manager, nothing is doing "proper" cleanup on it at that point. Not a huge deal in this specific case, but 🤷♂️.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org