You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/20 08:47:10 UTC

[GitHub] [airflow] surapuramakhil opened a new issue #17104: Resource usage in Airflow monitor metrix

surapuramakhil opened a new issue #17104:
URL: https://github.com/apache/airflow/issues/17104


   <!--
   
   Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
   Don't worry if they're not all applicable; just try to include what you can :-)
   
   If you need to include code snippets or logs, please put them in fenced code
   blocks.  If they're super-long, please use the details tag like
   <details><summary>super-long log</summary> lots of stuff </details>
   
   Please delete these comment blocks before submitting the issue.
   
   -->
   
   **Description**
   
   Airflow, as a task / workflow management tool, states execution times at task and dag level. But it lacks resource usage information.
   
   Like memory / Network / CPU usage, its max values at task level and DAG level
   
   <!-- A short description of your feature -->
   
   **Use case / motivation**
   
   <!-- What do you want to happen?
   
   Rather than telling us how you might implement this solution, try to take a
   step back and describe what you are trying to achieve.
   
   -->
   
   
   
   **Are you willing to submit a PR?**
   
   <!--- We accept contributions! -->
   
   **Related Issues**
   
   <!-- Is there currently another issue associated with this? -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] andormarkus commented on issue #17104: Resource usage in Airflow monitor metrix

Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #17104:
URL: https://github.com/apache/airflow/issues/17104#issuecomment-883255494


   Hi @surapuramakhil 
   
    - Kubernetes executor: these metrics are available on task level through Prometheus or your property monitoring provider
    - Celery executor: as I know, celery does not expose these metrics for each task.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] andormarkus commented on issue #17104: Resource usage in Airflow monitor metrix

Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #17104:
URL: https://github.com/apache/airflow/issues/17104#issuecomment-883255494


   Hi @surapuramakhil 
   
    - Kubernetes executor: these metrics are available on task level through Prometheus or your property monitoring provider
    - Celery executor: as I know, celery does not expose these metrics for each task.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #17104: Resource usage in Airflow monitor metrix

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17104:
URL: https://github.com/apache/airflow/issues/17104


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #17104: Resource usage in Airflow monitor metrix

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17104:
URL: https://github.com/apache/airflow/issues/17104#issuecomment-883580028


   Resource monitoring is not really part of airflow -it's outside of the domain of Airflow. You could bring that under discussion at the devlist if you want to ask for opinion of the community https://airflow.apache.org/community/ but this is whole new big feature that would drag the focus of Airflow from being great at "scheduling" and not "resource management"
   
   As mentioned by @andormarkus K8S provides good information already. Regarding the Celery, it makes no sense to have "per task" metrics, because Celery reuses running workers to run multiple tasks and they share resources, you can have all the insights you want into running Celery via Flower (when you use the Official Helm Chart from the Airflow community, flower is automatically enabled when you run Celery or CeleryKubernetesExecutor) - see https://airflow.apache.org/docs/helm-chart/stable/index.html
   
   For now this one is "close/won't do" - but again, if you have good arguments you can start thread at the devlist.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org