[GitHub] [airflow] Sinsin1367 edited a comment on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…

[GitBox <gi...@apache.org>]

Sinsin1367 edited a comment on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…
URL: https://github.com/apache/airflow/pull/6261#issuecomment-543251695
 
 
   > My initial thought: Just use prometheus/cadvisor for this. There are other tools that are better suited to capturing metrics than Airflow.
   > 
   > What about the KubeExecutor? If (and it's still an if to me) we are to add this it should work with pods launched by the executor too.
   
   The reason for fetching usage in this feature is to report the usage interleaved with application logs. This way the user would know at what stage in their job their usage is going up or down and will watch the usage as it is going. Also it eliminates necessity of using metric monitoring frameworks. Like in our case we use datadog for our metric monitoring which is not that easy to integrate in this case. I was hoping having this as an option that is turned off by default would not hurt any existing use-cases and adds the value of getting resource usage live in the logs to know exactly what events might be responsible for a bump in usage.
   
   KubeExecutor uses PodLauncher to launch pods also. However it seems it calls the async method of podLauncher which means it would not log.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services