You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/17 23:49:02 UTC
[GitHub] [airflow] didip opened a new issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
didip opened a new issue #13145:
URL: https://github.com/apache/airflow/issues/13145
**Description**
The Gantt chart contains a very valuable information: when and how long each task took serially.
For the purpose of monitoring and alerting, it would be nice if Airflow can do either of these:
* Expose the Gantt chart as an API so that DevOps people can write an alerting script using this API.
* Or allow users to configure alerting from inside the UI using the Gantt chart information.
**Use case / motivation**
As a user, I'd like to be able to alert to PagerDuty using information contained in the Gantt chart.
**Are you willing to submit a PR?**
I think it's a little hard for me to contribute.
**Related Issues**
None
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756851662
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-747776928
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip edited a comment on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip edited a comment on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756937687
What about something like this?
```
GET /dags/{dag_id}/dagRuns/{dag_run_id}/gantt
```
```
// Response
{
tasks: [
{
name: begin1,
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1
tasks: [
{
name: middle1
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1,
tasks: []
},
{
name: middle1
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 2,
tasks: []
}
]
},
{
name: begin2
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1,
tasks: [
{
name: middle2
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1,
tasks: [
{
name: end2
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1,
tasks: []
}
]
},
]
},
]
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756851662
@didip In my opinion, you can still make some specification suggestions. I see the benefits of such an API, but the details are easiest to discuss if we have the specifications. My previous comment on the specs got thumbs up from a few PMC members, so in my opinion we can discuss further improvements to the spec or some other way to address this issue, e.g. adding a sample code snippet to the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-757484026
> What about something like this?
We don't know about all the tries in the database, only about latest.
> taskName1: [
Using a user input as a key is very problematic as it hampers API development. You can't add a new key if needed without breaking changes.
For example: You can represent tags as a simple list of string
```json
"tags": ["AA", "BB"]
```
However, this means that you will not be able to add color to the tag if necessary.
A better APi should therefore look like this.
```json
"tags": [{"name": "AA", "name": "BB"}]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756851662
@didip In my opinion, you can still make some specification suggestions. I see the benefits of such an API, but the details are easiest to discuss if we have the specifications. My previous comment on the specs got thumbs up from a few PMC members, so in my opinion we can discuss further improvements to the spec or some other way to address this issue, e.g. adding a sample code snippet to the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13145:
URL: https://github.com/apache/airflow/issues/13145
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-753951469
@Dr-Denzy Could you please prepare draft specification first? When we determine the appearance of specific endpoints, we will be able to proceed to implementation. I'm afraid this API can be a bit more problematic than the other endpoints because it doesn't fit easily into REST. I omitted such an endpoint on purpose, because similar data can be obtained using the Batch API, but we can add it if necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756723786
Echo'ing @Dr-Denzy statement -- I agree the TaskInstance endpoint should contain all the information you need.
Please do let us know if you have any requirements that it does not fulfill and we can look into it again. For now I am closing this issue but we can reopen it if there are requirements that are not fulfilled by TI endpoint
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
Dr-Denzy commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-753940020
I will like to work on this task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756845232
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
Dr-Denzy commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-754112686
@didip apart from `dag_id` and `dag_run_id` parameters, what other parameters would suggest for this proposed Gantt chart API?
see: https://airflow.apache.org/docs/apache-airflow/2.0.0/stable-rest-api-ref.html#tag/TaskInstance
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756937687
What about something like this?
```
GET /dags/{dag_id}/dagRuns/{dag_run_id}/gantt
```
```
// Response
{
tasks: [
{
name: begin1,
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1
tasks: [
{
name: middle1
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1,
tasks: []
},
{
name: middle1
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 2,
tasks: []
}
]
},
{
name: begin2
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1,
tasks: [
{
name: middle2
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1,
tasks: [
name: end2
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1,
tasks: []
]
},
]
},
]
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip edited a comment on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip edited a comment on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756937687
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
Dr-Denzy commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-754110877
> @Dr-Denzy Could you please prepare draft specification first? When we determine the appearance of specific endpoints, we will be able to proceed to implementation. I'm afraid this API can be a bit more problematic than the other endpoints because it doesn't fit easily into REST. I omitted such an endpoint on purpose, because similar data can be obtained using the Batch API, but we can add it if necessary.
Ok. Noted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756851662
@didip In my opinion, you can still make some specification suggestions. I see the benefits of such an API, but the details are easiest to discuss if we have the specifications. My previous comment on the specs got thumbs up from a few PMC members, so in my opinion we can discuss further improvements to the spec or some other way to address this issue, e.g. adding a sample code snippet to the documentation.
Don't worry, everything will be fine and we'll find a solution that suits everyone.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip edited a comment on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip edited a comment on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756937687
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-748498177
I think having some Airflow metrics exposed via API is a good idea. I tried doing something with it - not via API but in webserver - #11632
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13145:
URL: https://github.com/apache/airflow/issues/13145
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy edited a comment on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
Dr-Denzy edited a comment on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-754112686
@didip apart from `dag_id` and `dag_run_id` parameters, what other parameters would you suggest for this proposed Gantt chart API?
see: https://airflow.apache.org/docs/apache-airflow/2.0.0/stable-rest-api-ref.html#tag/TaskInstance
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-757778998
> When it comes to showing all the retries, how did the UI do it?
This information is not available in the Web UI either.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-757570220
When it comes to showing all the retries, how did the UI do it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756845232
Sorry for the late response, folks. Life has been hectic recently.
@Dr-Denzy Yes, looking around other Airflow REST API, it looks like dag_id and dag_run_id are the only requirements.
If someone would be kind enough to reopen this issue, I'd greatly appreciate it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756940850
Second proposal, much flatter:
```
GET /dags/{dag_id}/dagRuns/{dag_run_id}/gantt
```
```
// Response
{
tasks: {
taskName1: [
{
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1
},
{
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 2
},
],
taskName2: [
{
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1
},
{
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 2
},
{
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 3
},
],
taskName3: [
{
start: unix-timestamp,
end: unix-timestamp,
duration: in-seconds,
try_number: 1
}
]
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756855831
> Sorry for the late response, folks. Life has been hectic recently.
> @Dr-Denzy Yes, looking around other Airflow REST API, it looks like dag_id and dag_run_id are the only requirements.
>
> If someone would be kind enough to reopen this issue, I'd greatly appreciate it.
Hi @didip -- Can you elaborate further like @mik-laj said. Even for Gantt Chart in Webserver need dag_id and dag_run_id:
So I am still not sure what else is required, thanks for the patience.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756851662
@didip In my opinion, you can still make some specification suggestions. I see the benefits of such an API, but the details are easiest to discuss if we have the specifications. My previous comment on the specs got thumbs up from a few PMC members, so in my opinion we can discuss further improvements to the spec or some other way to address this issue, e.g. adding a sample code snippet to the documentation.
Don't worry, everything will be fine and we'll find a solution that suits everyone. Thank you so much for wanting to work on this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] didip commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
didip commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-758091921
I wished I could take a screenshot of our deployment but that is not allowed at my company.
That said, I do see the reruns on our gantt chart UI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13145: Expose Gantt chart as an API or able to configure alerting using the Gantt chart.
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13145:
URL: https://github.com/apache/airflow/issues/13145#issuecomment-756723786
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org