You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nivdror (via GitHub)" <gi...@apache.org> on 2024/05/30 09:19:12 UTC
[I] Downstream sensors button [airflow]
nivdror opened a new issue, #39939:
URL: https://github.com/apache/airflow/issues/39939
### Description
Hi,
We have custom sensors that are based on BaseSensorOperator to link between our dags in our Airlow environment.
Right now we have the ability to move backwards via external DAG button when pressing on the sensor in the graph UI.
This is not enough, I would to be able to travel forward as well. I would like the graph to show on which tasks a sensor is listening and tab of connection(with links of course) to those sensors.
We created this feature in our own old airflow but I don't see why this cannot be in apache airflow.
Here is an example:
after pressing the downstream sensors button
### Use case/motivation
I'm trying to achieve a better way to move/investigate dags in airflow
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2258956878
We did a bit to show datasets connected upstream and downstream of a DAG on the DAG page. It's not a bad idea to include all types of dependencies like sensors and triggers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2139209916
If you would like propose a feature better to provide a PR, otherwise it might take unpredictable time to implement it. In short until someone who want to implement this and might take a time from couple day to decades.
Apache Airflow is an OSS and everyone could propose changes by raising Pull Request, and if it pass code review it would be include into the next minor version of Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "nivdror (via GitHub)" <gi...@apache.org>.
nivdror commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2858776526
obviously, I'm not using airflow 3.0 yet, but I understand not wanting to advance 2.10 further.
The main idea of this feature is to allow navigation between adjacent dags.
I don't think the new feature will get there but once we moved airflow 3.* maybe I will create another PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2139764488
Ok, well, we'll leave this as an open feature request but it might not get any traction without someone else seeing the value and making it happen. Maybe you could try to build some connections with other UI interested community members and collaborating from there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "nivdror (via GitHub)" <gi...@apache.org>.
nivdror commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2140546076
Hi, we won't use datasets.
Our architecture is built mainly on three layers - scheduler layer, context layer and compute layer.
Each got it's dependencies and definition. So we see airflow as scheduler which doesn't need to what it's running(spark, MapReduce etc') nor where (CDH, EMR, Databricks etc') nor the data artifacts we output.
This helps us migrate through different providers of Airflow + plus helps us decouple the dependencies.
In summary, we don't put all of our eggs in one basket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "nivdror (via GitHub)" <gi...@apache.org>.
nivdror commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2857463099
@bbovenzi Hi,
how is this a duplicate of the https://github.com/apache/airflow/issues/49576 PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2139171112
> Are you willing to submit a PR?
> [ ] Yes I am willing to submit a PR!
> We created this feature in our own old airflow but I don't see why this cannot be in apache airflow.
🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "nivdror (via GitHub)" <gi...@apache.org>.
nivdror commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2139233684
I'm aware that Airflow is a open source project. However I don't possess any frontend skills nor the time to learn them.
So I open a feature request in hope that someone will think the idea is worthy and implement it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "nivdror (via GitHub)" <gi...@apache.org>.
nivdror commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2139731709
@nathadfield No, those people have left the company
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2140114111
Isn't that exactly what you can see in the dataset view? You can move upstream or downstream
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "nivdror (via GitHub)" <gi...@apache.org>.
nivdror commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2139771669
@nathadfield thank you. I will see what I can do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2855391215
Duplicate of https://github.com/apache/airflow/issues/49576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2139586372
@nivdror Maybe you could collaborate with the people who did create this as a custom feature to bring it to the project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi closed issue #39939: Downstream sensors button
URL: https://github.com/apache/airflow/issues/39939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downstream sensors button [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #39939:
URL: https://github.com/apache/airflow/issues/39939#issuecomment-2858504514
> [@bbovenzi](https://github.com/bbovenzi) Hi, how is this a duplicate of the [#49576](https://github.com/apache/airflow/issues/49576) PR?
With the launch of 3.0, we won't be adding any new features to 2.10. But that issue is to better incorporate sensors and triggers in the dag graphs. If there is something missing then please chime in.
I figure that task.doc_md and extra links with a better graph should get you most of the way there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org