[jira] [Created] (AIRFLOW-3545) Can't use Prometheus or other pull based instrumentation systems to monitor Tasks launched on Kubernetes

["Aditya Vishwakarma (JIRA)" <ji...@apache.org>]

Aditya Vishwakarma created AIRFLOW-3545:
-------------------------------------------

             Summary: Can't use Prometheus or other pull based instrumentation systems to monitor Tasks launched on Kubernetes
                 Key: AIRFLOW-3545
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3545
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Aditya Vishwakarma


Prometheus, which is a common way to instrument services on Kubernetes, uses a pull based mechanism to fetch metrics. Which generally involves a service exposing a `/metrics` endpoints. This endpoint is scraped every 30 secs by prometheus to collect metrics. 

This requires a port to be specified in the generated Pod config. Something like below.
{code:java}
// Sample Pod Spec
apiVersion: v1
kind: Job
metadata:
  name: batch-job
spec:
  ports:
  - name: metrics
    port: 9091 # port to fetch metrics from
    protocol: TCP
    targetPort: 9091
{code}
Currently KubernetesPodOperator doesn't have any options to open ports like this.

Is it possible to have an option to do this?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)