You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "js661818 (via GitHub)" <gi...@apache.org> on 2023/03/03 10:47:10 UTC
[GitHub] [airflow] js661818 opened a new issue, #29897: Import error of new dags cannot be show on web ui for common users
js661818 opened a new issue, #29897:
URL: https://github.com/apache/airflow/issues/29897
### Apache Airflow version
2.5.1
### What happened
When we put a new dag in airflow and the dag contains some error which cannot be parsed by the airflow scheduler,
* The common role user is assigned comm role which has the permission of **read Import Error**
* Expectation: we expect thec should be seen in ui by the dag's user(controlled by access_control)
* Result : We cannot see the **import error** by common Role user(only admin can see it )
- admin see this:
- common user see no dag or **import error**
### What you think should happen instead
_No response_
### How to reproduce
Just use a error new dag which grants the permission to common user not a admin
### Operating System
Centos7
### Versions of Apache Airflow Providers
2.5.0
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-5942898854
This was fixed by #61163, released in 3.1.7: import errors for files that have no registered Dag yet are now returned to non-admin users who can view import errors. On main, #69790 moves these behind a dedicated import errors permission that admins can grant to other roles. If you still see this on the latest release, comment here and we'll reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1970616571
No idea. If someone provide more details, mayb someone will look at it - not sure why tagging me would help here. Ideally - if someone adds more information and logs here, there is a chance somoene will diagnose and fix it. I think everyone's case is different - dpending on the conrete set of permissions and ways how their dags have permissions set the access might be possible or not.
So describing more details about what permissions your users have, what permissions your roles have and what permissions your dags have, might help someone to solve the issues - note that we had recently a few security fixes that LIMITED permissions to dag imports because they were too broad. So I suggest to take a close look at your permissions and see if the user has the right set of permissions (and I cannot have an easy answer what permissions is the right set because if you have your own custom permissions and roles set (as was in the original issue) - it's up to you to build them in the way that makes sense for you.
Also if you turn on debug logs - you might be able to see why the user cannot see the dag. So if you gather all the information, permission configuration and logs - it **might** help someone to investigate it
Also - yes - there were some fixes since 2.6 to that, so its's likely 2.8.2 is more accurate about it - but if you have custom set of permissions, then you need to make sure they are as you expect them to have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "ananth-sabhapathi (via GitHub)" <gi...@apache.org>.
ananth-sabhapathi commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1799210177
So this is an issue with the latest version too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29897: Import error of new dags cannot be show on web ui for common users
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1454850546
Feel free to take a stab on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "ahipp13 (via GitHub)" <gi...@apache.org>.
ahipp13 commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1779790643
This is still open in 2.6.3. Commenting here because I want to know if there are any updates on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ananth-sabhapathi commented on issue #29897: Import error of new dags cannot be show on web ui for common users
Posted by "ananth-sabhapathi (via GitHub)" <gi...@apache.org>.
ananth-sabhapathi commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1677889928
Yes, @potiuk . Still we have this issue..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "ananth-sabhapathi (via GitHub)" <gi...@apache.org>.
ananth-sabhapathi commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2672164860
This is still an open issue in v2.10.5. Just tested and still custom role without "can read on DAGs" is not showing the import error. "can read on DAGs" -> Shows all teams DAGs to everybody.
Granting "can read on DAG <DAG_ID>" permission is not helping though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2769720809
> Is anyone working on this issue?
Definitely if you want you can, or maybe you can find someone who will ? I guess best thing is to also check Airflow 3 when it is out in month or so - and rc is planned this week, possibly that error is fixed there, basically most of the community is not going to make any more updates to Airlfow 2 except critical / security bugfixes, so checking how this looks like in Airflow 3 and upgrading might likely be the fastest way you can get it fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #29897: Import error of new dags cannot be shown on web UI for common users
URL: https://github.com/apache/airflow/issues/29897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "wolfier (via GitHub)" <gi...@apache.org>.
wolfier commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2259556807
I am unable to replicate the behaviour in Airflow 2.5.1 or 2.6.3. In the original scenario, the custom role does not have the respective read on DAG permission.
The role must have all the following to see import errors of the respective DAG the role can read.
* can read on Website
* can read on DAG:<insert_dag_Id>
A role with the following permissions can see ALL import errors.
* can read on Website
* can read on DAGs
I believe the permission `can read on ImportError` does not impact whether a user can see the import errors in Airflow 2.5.1 or 2.6.3.
This is not true on Airflow 2.9.3.
The role must have all the following to see import errors of the respective DAG the role can read.
* can read on Website
* can read on ImportError
* can read on DAG:<insert_dag_Id>
The role must have all the following to see import errors to see all import errors DAG.
* can read on Website
* can read on ImportError
* can read on DAGs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "wolfier (via GitHub)" <gi...@apache.org>.
wolfier commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2261449407
> So if the DAG hasn't been uploaded yet, and it gets an ImportError
Do you mean if the DAG is present in the directory configured to be the dags folder but has never been successfully parsed? If that is the case, the DAG would never have been inserted in the `dags` table. The insertion only happens after a DagFileProcessorProcess has successfully parsed the file.
Even if the permission is created somehow or had existed before, if the DAG is not in the `dags` table, those permissions are meaningless.
---
I am able to replicate the behaviour only after truncating the `dag` table or deleting the entry of the associated DAG with the import error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29897: Import error of new dags cannot be show on web ui for common users
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1453333100
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-3986501259
@timlichen We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29897: Import error of new dags cannot be show on web ui for common users
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1680850873
feel free
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "ahipp13 (via GitHub)" <gi...@apache.org>.
ahipp13 commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2260894175
> I am unable to replicate the behaviour in Airflow 2.5.1 or 2.6.3. In the original scenario, the custom role does not have the respective read on DAG permission.
>
> The role must have all the following to see import errors of the respective DAG the role can read.
>
> * can read on Website
> * can read on DAG:<insert_dag_Id>
>
> A role with the following permissions can see ALL import errors.
>
> * can read on Website
> * can read on DAGs
>
> I believe the permission `can read on ImportError` does not impact whether a user can see the import errors in Airflow 2.5.1 or 2.6.3.
>
> This is not true on Airflow 2.9.3.
>
> The role must have all the following to see import errors of the respective DAG the role can read.
>
> * can read on Website
> * can read on ImportError
> * can read on DAG:<insert_dag_Id>
>
> The role must have all the following to see import errors to see all import errors DAG.
>
> * can read on Website
> * can read on ImportError
> * can read on DAGs
I am currently on 2.9.2, and this still exists. The reason being is because for one of my roles, the permission "can read on DAG:<insert_dag_Id>" is assigned inside the DAG itself. So if the DAG hasn't been uploaded yet, and it gets an ImportError, then that means it doesn't enter the UI, which means the permission "can read on DAG:<insert_dag_Id>" does not get assigned to the role, so they cannot see it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "andreyolv (via GitHub)" <gi...@apache.org>.
andreyolv commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2296580902
I also have this issue in version v2.9.1. In summary:
If the permissions 'can read on ImportError' and 'can read on DAG:test123' are configured directly on the role, the user can see the import error of this dag.
However, if the 'can read on ImportError' permission is configured in the role and the access_control={"Role": {"can_read"}} permission is configured in the dag test123, the user cannot see the import error of this dag.
I'm actually using dag_policy to mutate the access_control field, but I don't think that matters in this case. Example:
```
owner_list = ["AAA", "BBB", "CCC"]
dag_owner = dag.owner.split(",")[0]
if not dag_owner in owner_list:
raise AirflowClusterPolicyViolation(
f"DAG {dag.dag_id} has no correct owner. A correct owner {owner_list} is required."
)
else:
dag.access_control = {dag_owner: {"can_read"\, "can_edit"}}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "SujjithS (via GitHub)" <gi...@apache.org>.
SujjithS commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2768109785
Is anyone working on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "bmoon4 (via GitHub)" <gi...@apache.org>.
bmoon4 commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2294622876
Been observing this issue for long time ... and my team ended up building a dashboard(Grafana) which uses Airflow API calls to get import error data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] timlichen commented on issue #29897: Import error of new dags cannot be show on web ui for common users
Posted by "timlichen (via GitHub)" <gi...@apache.org>.
timlichen commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1680681639
Hey @potiuk, I'm interested in taking this on if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "timlichen (via GitHub)" <gi...@apache.org>.
timlichen commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1779820918
I was unable to replicate this error/behavior when I looked into this previously, it may no longer be an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "ahipp13 (via GitHub)" <gi...@apache.org>.
ahipp13 commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1779842177
@timlichen I am able to replicate this behavior with my setup. I have 2 roles:
TEAM_ROLE:
Normal User role:
We run on kubernetes with a git sync sidecar. When we upload a DAG to this sidecar for the first time with this in the access control:
access_control={'TEAM_ROLE': ['can_edit', 'can_read']}
The team role cannot see the import error:
But the user role can:
So this is still an issue. I am looking into a workaround where it just emails out if there is an import error as I do not have the time to write my very own custom security manager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "ahipp13 (via GitHub)" <gi...@apache.org>.
ahipp13 commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-2262953586
> > So if the DAG hasn't been uploaded yet, and it gets an ImportError
>
> Do you mean if the DAG is present in the directory configured to be the dags folder but has never been successfully parsed? If that is the case, the DAG would never have been inserted in the `dags` table. The insertion only happens after a DagFileProcessorProcess has successfully parsed the file.
>
> Even if the permission is created somehow or had existed before, if the DAG is not in the `dags` table, those permissions are meaningless.
>
> I am able to replicate the behaviour only after truncating the `dag` table or deleting the entry of the associated DAG with the import error.
>
> According to the [source code](https://github.com/apache/airflow/blob/main/airflow/api_connexion/endpoints/import_error_endpoint.py#L94-L99), this is expected behaviour. If the DAG entry does not exist in the `dag` table, the query will never return the DAG with the import error even it is selecting the DAG. A user with the Admin role can see the import error because the query result is not filtered.
@wolfier yes that makes total sense, its just our use case requires us to do it this way so we run into this quite a lot... We have 2 teams using our Airflow instances, and we don't want them to see each other's DAG's. So the only way we found to do this is to set the permissions within the DAGs themselves.
Just an idea for you or anybody else that comes across this. Would it be possible to be able to use regular expressions for permissions? For example, the 2 teams that use the Airflow instance always use the same prefix (their team name) so the DAG names are always {team_name}_{dag_name}. Could you add functionality to be able to do a permission like "can read on DAG:teamname.*" to give the role the ability to read all DAGs that start with "teamname"?
Just an idea to throw out there. If I get some free time I can try and explore this more but wanted to get it out there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import error of new dags cannot be shown on web UI for common users [airflow]
Posted by "priya369 (via GitHub)" <gi...@apache.org>.
priya369 commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1970489250
@potiuk still the issue is there in 2.6.3 version,any update on fix of this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ananth-sabhapathi commented on issue #29897: Import error of new dags cannot be show on web ui for common users
Posted by "ananth-sabhapathi (via GitHub)" <gi...@apache.org>.
ananth-sabhapathi commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1674095680
Is this still a open issue in 2.6.3 version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29897: Import error of new dags cannot be show on web ui for common users
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29897:
URL: https://github.com/apache/airflow/issues/29897#issuecomment-1676475490
> Is this still a open issue in 2.6.3 version?
Can you check it and let us know please @ananth-sabhapathi ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org