[jira] [Commented] (AIRFLOW-3545) Can't use Prometheus or other pull based instrumentation systems to monitor Tasks launched on Kubernetes

["Ash Berlin-Taylor (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726763#comment-16726763 ] 

Ash Berlin-Taylor commented on AIRFLOW-3545:
--------------------------------------------

Right now this is not possible, and due to the heavy use of (sub)processes by Airflow this isn't a trivial change.

I'd suggest taking a look at https://github.com/prometheus/statsd_exporter

> Can't use Prometheus or other pull based instrumentation systems to monitor Tasks launched on Kubernetes
> --------------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-3545
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3545
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Aditya Vishwakarma
>            Priority: Major
>
> Prometheus, which is a common way to instrument services on Kubernetes, uses a pull based mechanism to fetch metrics. This  involves a service exposing a `/metrics` endpoint. This endpoint is scraped every 30 secs by prometheus to collect metrics.
> This requires a port to be specified in the generated Pod config. Something like below.
> {code:java}
> // Sample Pod Spec
> apiVersion: v1
> kind: Job
> metadata:
>   name: batch-job
> spec:
>   ports:
>   - name: metrics
>     port: 9091 # port to fetch metrics from
>     protocol: TCP
>     targetPort: 9091
> {code}
> Currently KubernetesPodOperator doesn't have any options to open ports like this.
> Is it possible to have an option to do this?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)