You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/15 15:46:33 UTC
[GitHub] [airflow] cherusk opened a new issue #11549: Operator for prometheus API interaction
cherusk opened a new issue #11549:
URL: https://github.com/apache/airflow/issues/11549
**Description**
Instead of building prometheus queries on other operators, e.g. python operator, come forth with a dedicated prometheus API interacting operator.
I'm prepared to substantially contribute to such a operator and would mostly like to learn would that be acceptable for the AIRFLOW community.
**Use case / motivation**
For a project of mine, godon[1], which is basing on airflow at its core I'm seeking to have an operator that interacts with the prometheus API to at least gather metrics. There are already prometheus-api shim libarires[2] offered which makes a dedicated operator quite tangible.
[1] https://github.com/cherusk/godon
[2] https://github.com/AICoE/prometheus-api-client-python
**Related Issues**
Not knowingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-712179086
We are open to all operators, but they must have unit tests. :-)
Related: https://github.com/apache/airflow/issues/10386
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-709415662
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-736808799
Oh sorry, yes. I never intended to close this!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cherusk commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
cherusk commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-712219340
> Could you provide some use cases for such an operator?
I provided that of godon, where it would scrape the exporters or aggregator prometheus instance periodically in an optimization DAG. Would you need more detail?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-736858766
I think we need to discuss this in the devlist. What is the future of Airflow in terms of new "providers" - whether we should incorporate more providers in Airflow after 2.0 and support them by community (and possibly grow it by accepting more committers from different areas) or whether we should slim it down and not accept wider contributions).
This has never been discussed openly - and while I feel there are some opinions there, there, they are just that - opinions, until we make it a point and discuss in the community and make decisions.
I do not know personally what is my opinion at that because it is exactly this - it has not been discussed, neither pros/cons nor actual approach here has ever been discussed in the devlist as far as I can remember.
@ashb -> I think it would be great if you start this discussion at the devlist, before we lightly discard such ideas whether new providers will be added in the community or as 3rd-party implementations ?
I think, whatever decision will be made, it should be made after raising the issue and public discussion at the devlist.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-737200953
@potiuk I wasn't intending to create any consensus or precedent here -- merely pointing out that "operators are just python modules" so can easily live outside of the Airflow tree.
This is not new behaviour, nor should it controversial, as operators are just Python classes loaded when parsing the DAG file -- and we'd have to go out of our way in Airflow to stop this being the case.
I'll have a think if I _do_ want to create such a precedent/proposal.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-738125044
I think this is something definitely after we release 2.0 - the long planned "What's beyond 2.0" discussion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-736788563
It can be yes, but it can also be done by anyone as a third party module, and then it doesn't need a small handful of busy people to review, test, do release management and follow Apache policies around releases.
I have no problem with it being in tree, but out of tree does have advantages for moving quickly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-712179086
We are open to all operators, but they must have unit tests. :-) Prometheus is a popular technology among contributors, so I think it shouldn't be a problem to find the right reviewer.
Related: https://github.com/apache/airflow/issues/10386
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cherusk commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
cherusk commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-736794938
Understand your point and it's fair. Yet, I see no harm in leaving this open, as it's declared as what it is, a feature idea and should stay open to the world in that context. If mainliners don't want to support it, so be it. But closing it would signal wrong impression, at least for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cherusk commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
cherusk commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-736783817
Why, it's a valid feature request which can be actioned by me or others
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-736564054
It is also possible that you can create the operator "externally" to the Airflow source code -- https://github.com/great-expectations/airflow-provider-great-expectations is one such example.
Then if it proves popular we could discuss moving it "in tree" if needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11549: Operator for prometheus API interaction
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11549:
URL: https://github.com/apache/airflow/issues/11549#issuecomment-712180111
Could you provide some use cases for such an operator?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org