You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "JonnyWaffles (via GitHub)" <gi...@apache.org> on 2023/05/25 14:31:36 UTC
[GitHub] [airflow] JonnyWaffles opened a new issue, #31544: UI TaskInstance Context Menu Missing?
JonnyWaffles opened a new issue, #31544:
URL: https://github.com/apache/airflow/issues/31544
### Apache Airflow version
2.6.1
### What happened
Hi team, we just upgraded to 2.6.1 and now I cannot find the TaskInstance Context menu (detailed [here](https://airflow.apache.org/docs/apache-airflow/stable/ui.html#task-instance-context-menu) in the docs). In the new UI, how do I force a task to run via web ui and ignore any dependencies?
### What you think should happen instead
_No response_
### How to reproduce
Open the UI click on a task instance, try to find the old Task Actions interface.
### Operating System
ubuntu20.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Possibly add a feature to "Run Tasks" from the UI [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-3824511754
Yes, that doesn't replace the requested feature but is more a workaround in the meantime since nobody seems to actively be working on this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Adaverse commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563080479
Since you want to re run the task instance. You can clear the task instance and unselect the default selected options recursive and downstream (i believe) and clear. It should re run the task without affecting the deps.
PN - its not a bug. Its removed on purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Adaverse commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563105262
Will update it with all the new UI screens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JonnyWaffles commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "JonnyWaffles (via GitHub)" <gi...@apache.org>.
JonnyWaffles commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563285646
Right, so is it intended to remove the `Run` option entirely from the UI? Without Run I don't have a way to force trigger execution without using the CLI. Our environment is locked down so the CLI isn't an option. I need to use the UI to manually force a task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563403987
@JonnyWaffles Yes, it was removed in https://github.com/apache/airflow/pull/29706 because it was not working properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Possibly add a feature to "Run Tasks" from the UI [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-2842088725
> . So providing a button that in essence just moves the tasks to the front of the queue or changing the priority to something very high
Airflow supports [priority weight](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/priority-weight.html) to do exactly that. (move task to the front of the queue). You can also take a look a pools
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JonnyWaffles commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "JonnyWaffles (via GitHub)" <gi...@apache.org>.
JonnyWaffles commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1564391829
Okay so is there any other UI driven alternative to using the CLI to "force run" (-f -A -i) a task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1564640140
I don't think so. I believe the idea was to use `clear task` to rerun a task, but this is not equivalent to the run options we had before.
We can convert this into a feature request, the goal would be to re-implement the `run task` but addressing the original issue in #29706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Possibly add a feature to "Run Tasks" from the UI [airflow]
Posted by "notatallshaw-gts (via GitHub)" <gi...@apache.org>.
notatallshaw-gts commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-3785436695
+1 to this feature request, I recently upgraded from Airflow 2.3.4 to Airflow 2.11.0 and I hit this scenario:
1. A ShorCurcuitOperator skipped downstream tasks, based on some tasks skipping certain holidays
2. But I want to manually run one of the immediate downstream tasks
3. Clearing the the task does not work because the task checks if it's in the upstreams `skipmixin_key` xcom list
So to run the task I need to either have permissions to clear the xcom from the ShorCurcuitOperator, run the Airflow CLI, or enforce a rule on all my DAGs that nothing but an empty operator should depend directly on a ShorCurcuitOperator.
I understand the technical reasons the old force run feature was remove, but this is a common use case regression for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Possibly add a feature to "Run Tasks" from the UI [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-3823783637
I guess you could mark the upstream tasks as 'success' and manually update the xcom values to the appropriate value for your task to run when it's marked as 'clear'.
Editing xcom will be possible through the UI in 3.2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] harinipriya-nv commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "harinipriya-nv (via GitHub)" <gi...@apache.org>.
harinipriya-nv commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1663348350
Hi All , We are also looking for the "run task" feature from UI. Can you please let me know if there is update on this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563238242
The actions still exist for the graph and gantt views, we just dropped "Run":
<img width="617" alt="Screenshot 2023-05-25 at 1 05 40 PM" src="https://github.com/apache/airflow/assets/4600967/8b1604d1-6ea8-471a-9a2f-ab22e74a1bc7">
Otherwise, in the grid view the actions are in the details nav:
<img width="1051" alt="Screenshot 2023-05-25 at 1 06 05 PM" src="https://github.com/apache/airflow/assets/4600967/56e013a2-6e69-4e41-9194-17d5ce6c8cdd">
We should update the docs to reflect that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Possibly add a feature to "Run Tasks" from the UI [airflow]
Posted by "brs-fcoo (via GitHub)" <gi...@apache.org>.
brs-fcoo commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-2839906085
I know this is a quite old issue, but since it's still open and my request is related I didn't want to open a new issue.
We are migrating from an old Airflow setup to a new one, and found the 'run' button to be missing as well.
It is not a feature that we have used often, but a few times it has been very handy as we are running a very complex setup that includes a HPC cluster. Here we sometimes have used the feature on tasks which are in queued state, that we would actually like to be run immediately.
So my request is not to bring back the old functionality as it was, and I agree that it probably was a good thing to remove as it was implemented. Our use case is to run a task, which is in a state where it is ready to run but just queued. So providing a button that in essence just moves the tasks to the front of the queue or changing the priority to something very high, would be very helpful for us and as I see it without introducing any security concerns as we are not modifying the state directly or ignoring dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563110411
@Adaverse assigned you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JonnyWaffles commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "JonnyWaffles (via GitHub)" <gi...@apache.org>.
JonnyWaffles commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563180604
> Since you want to re run the task instance. You can clear the task instance and unselect the default selected options recursive and downstream (i believe) and clear. It should re run the task without affecting the deps.
>
> PN - its not a bug. Its removed on purpos
Hi @Adaverse, I want to clear it and do a force rerun ignoring any dependencies and or skips. Is this action still available? It's not enough to just clear it since a down stream task is skipping. In the old UI I could choose ignore everything and run it similar to using the CLI flags -f -A -i.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Possibly add a feature to "Run Tasks" from the UI [airflow]
Posted by "notatallshaw-gts (via GitHub)" <gi...@apache.org>.
notatallshaw-gts commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-3824418123
@pierrejeambrun That's a slightly better workaround, but still a terrible UX for users of the WebUI, it involves:
1. Finding the upstream task that skipped it (there could be dozens of upstream tasks, including multiple operators that could or did skip the operator)
2. Understanding the internal details of how short circuit operators work
3. Editing in a value that could easily introduce manual errors
Even once that workaround is available, I would still be +1 to adding this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1663390138
Nope. The "run task" has been removed from the UI, also because it was a potential security issue, we will not get into details of that but a CVE announcement about it will be done soon (watch users@ and ASF announcements)
So the update here is - for now you can use clear task. Running single task from the UI is not likely to come back in its past form. I will turn it into feature request and I will encourage any one who wants somethign similar to explain their use case and why they cannot use any of the alternatives, including CLI commands (primary CLI commands were foreseen for such things like testing and running individual tasks).
Running single task on the UI and using webserver is not compliant with Airfow security model https://airflow.apache.org/docs/apache-airflow/stable/security/index.html, where users of the UI should not get the capabilities of running arbitrary code coming from the DAGs on the webserver.
AIrflow CLI on the other hand, with access to the Airflow servers is the place where running arbitrary code is expected and you have authorisation to do so. If it is just a "convenience" and "bypassing" of the security perimeter we set then it's not likely to come back as user feature and you shoudl switch to CLI and runnign any kind of tests and tasks there if you need. If there is a legitmate feature, the please explain in detail the use case you have and why CLI access is not enough @JonnyWaffles @harinipriya-nv.
Hopefully if you do it in a clear an concise way and explain the use case, somebody will implement it. But this is like with all things in open source. If there is no-one doing it, it won't get implemented. Explaining the feature you want and reasoning why it is needed is the next best thing after proposing a PR doing it that you can do to increase the likelihood someone will pick an interest and implement it
Converting it into a feature request and waiting for the detailed explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563187155
@JonnyWaffles Indeed, it appears that this is no longer possible, which seems to be a bug if I'm not mistaken.
@bbovenzi @pierrejeambrun any idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #31544: UI TaskInstance Context Menu Missing?
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1563090658
As @Adaverse explained, it's still possible using the clear button, but the documentation is not up to date and needs to be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on issue #31544: Possibly add a feature to "Run Tasks" from the UI
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1741729287
The issue was not with the nature of the feature but because of a bad implementation, allowing wrong access control vulnerability.
If implemented properly I don’t see any problem having the Run feature back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JonnyWaffles commented on issue #31544: Possibly add a feature to "Run Tasks" from the UI
Posted by "JonnyWaffles (via GitHub)" <gi...@apache.org>.
JonnyWaffles commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-1738010381
Hi team, apologies for the delayed response @potiuk!
After reading through the security model, I don't really understand how force running a task is non-compliant, but then I'm no security engineer. Force running a task feels very similar in most use cases to clearing and letting the existing schedule rerun. We've always used it for administrative "break glass", where a single task may need to be executed regardless of its dependencies due to some external circumstances.
Our environment is extremely locked down, so the UI is the **only real** way to execute administrative tasks outside of a deployment pipeline. Even administrators cannot assume the Airflow identity required by CLI, because all production credentials are effectively machine read only. Only SSO, backed by multi-factor, login enables administration tasks. This feels appropriate to me because
I am definitely willing to work on this feature once it is prioritized by our internal team, but in the immediacy clearing is good enough, but I certainly miss the legacy feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Possibly add a feature to "Run Tasks" from the UI [airflow]
Posted by "brs-fcoo (via GitHub)" <gi...@apache.org>.
brs-fcoo commented on issue #31544:
URL: https://github.com/apache/airflow/issues/31544#issuecomment-2843050983
The issue isn't the general priority weight, that is set as desired, but a specific (not normal) situation might occur where we would like to have a task jump the queue (e.g. change the priority for that single task instance).
The challenge for us, where the run button has been appreciated, is e.g. after serious downtime like a day or so, where we have to catch up with our forecasts (ocean and weather) and products.
To give a specific example (real case):
We have a big DAG that can have 3 simultaneous DAG runs (under normal circumstances there will not be more than one at a time). After a power outtage we are 6 DAG runs behind. As there a maybe 100GB data to download and process there a many tasks (thousands) to be run and we end up having a single task i the first DAG run queued for a long time as the same tasks in the two other runs of course have the same priority weight, which causes the fourth DAG run to be halted. He we have used the 'run' button to manually coerce a few tasks to run immediately so the DAG run can be finished, and the next one can start (downloading a lot of data). This is of course just 1 of about 30 DAGs which also needs to catch up.
Just to be clear, everything is working as intended and our Airflow setup runs like a charm, but would just take longer if we didn't use the run button as a single task can block a whole DAG run from starting. So this is just an example where we have seen the benefit of being able to jump queue.
Our setup is complex and could be designed differently, but I believe that no matter how we do similar situations would happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org