[jira] [Comment Edited] (AIRFLOW-1310) Kubernetes execute operator

["Michal Dziemianko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094330#comment-16094330 ] 

Michal Dziemianko edited comment on AIRFLOW-1310 at 7/20/17 8:10 AM:
---------------------------------------------------------------------

It was exactly the same sort of thinking on my side. 

Good point about the logs. It was sort of design decision as we capture all logs centrally in ELK so we had no need to stream the logs to airflow as such. It is not hard to add though. 

Also regarding your comment that it would be more powerful if you can supply a pod definition directly - thats another decision to hide the pod definition as such to make deployment of simple jobs simpler. I have a version where you can give a pod definition instead in case you need to do something complex, but for most jobs we have we just have an image (often with default command) and we dont want to force people to learn yet another library just to run a simple command. 

Do you mind collaborating to get out something nice and useful to everybody?


was (Author: mdziemianko):
It was exactly the same sort of thinking on my side. 

Good point about the logs. It was sort of design decision as we capture all logs centrally in ELK so we had no need to stream the logs to airflow as such. It is not hard to add though. 

Also regarding your comment that it would be more powerful if you can supply a pod definition directly - thats another decision to hide the pod definition as such to make deployment of simple jobs simpler. I have a version where you can give a pod definition instead in case you need to do something complex, but for most jobs we have we just have an image (often with default command) and we dont want to force people to learn yet another library. 

Do you mind collaborating to get out something nice and useful to everybody?

> Kubernetes execute operator
> ---------------------------
>
>                 Key: AIRFLOW-1310
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1310
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: operators
>            Reporter: Yu Ishikawa
>            Assignee: Dennis Docter
>
> We should support operators for Kubernetes to execute containers remotely.
> - Create a Kubernetes cluster
> - Execute PODs
> - https://github.com/kubernetes-incubator/client-python



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)