You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/18 23:59:18 UTC

[GitHub] [airflow] everglory99 opened a new issue #14307: `KubernetesPodOperator` raises exception when `pod_template_file` contains multiple objects

everglory99 opened a new issue #14307:
URL: https://github.com/apache/airflow/issues/14307


   In Airflow 2.0.1, we are trying to use `pod_template_file` in `KubernetesPodOperator`. The `pod_template_file` contains several sections specifying `ConfigMap`, `Service` and `Deployment` etc.. However, I am seeing this error from Airflow
   ```
   raise ComposerError("expected a single document in the stream",
   yaml.composer.ComposerError: expected a single document in the stream
   but found another document
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jedcunningham commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-813716813


   @everglory99, have you considered exposing a way to add these extra objects into your helm chart, say via `extraConfigMaps` etc or more generally `extraObjects`? It seems much saner to update them with helm instead of trying to deploy them with every task run (which is a whole different can of worms, unless you run with `parallelism` set to 1 😀).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] everglory99 commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
everglory99 commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-781733615


   The YAML file is generated by a helm chart our team developed by running `helm template`. The YAML file looks like
   ```
   ---
   apiVersion: v1
   kind: ConfigMap 
   metadata:
     name: dev-configmap
   
   data:
     SOME_KEY: "SOME_VALUE"
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: dev-engine
   spec:
     ports:
       - name: http
         port: 80
         protocol: TCP
     selector:
       app.kubernetes.io/name: engine
       app.kubernetes.io/instance: dev
   ---
   SOME MORE OBJECTS...
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] everglory99 commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
everglory99 commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-781750514


   Thanks @dimberman for the suggestions! I am wondering how to launch non-pod objects using Airflow? Should I just use K8s Python client?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dimberman closed issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
dimberman closed issue #14307:
URL: https://github.com/apache/airflow/issues/14307


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-814240786


   @everglory99 I'm going to close this but please feel free to reopen if you have more questions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-781734630


   @everglory99 I don't think you can do that with pod_template file. we expect just a single pod definition. You're trying to launch a pod with an attached service/configmap?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-809771575


   @everglory99 Yep! That should totally work. You can even build a custom operator/donate it if you think it will be useful :)
   
   @kaxil yes I think this is a feature request.
   
   @everglory99 would you be intereseted in building that? I can help get you setup with a dev environment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #14307: `KubernetesPodOperator` raises exception when `pod_template_file` contains multiple objects

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-781714103


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-781745623


   > Hmm so one option would be to create a subclass of the KubernetesPodOperator that takes in file paths for these extra objects and launches them before launching the pod with a pod_template_file that launches JUST the pod and then deletes those objects in a "finally" block to ensure they are gone after the task ends. Would that work?
   > 
   > cc: @kaxil
   
   @dimberman Would you consider this a feature request then instead of a bug?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-781742679


   Hmm so one option would be to create a subclass of the KubernetesPodOperator that takes in file paths for these extra objects and launches them before launching the pod with a pod_template_file that launches JUST the pod and then deletes those objects in a "finally" block to ensure they are gone after the task ends. Would that work?
   
   cc: @kaxil 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-814235468


   @jedcunningham good point. Deploying k8s objects in parallel can potentially lead to weird states.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] everglory99 commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
everglory99 commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-781737239


   Yes, that is exactly the use case here. Do you have any suggestions how to achieve this in Airflow? Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #14307: KubernetesPodOperator raises exception when pod_template_file contains multiple objects

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #14307:
URL: https://github.com/apache/airflow/issues/14307#issuecomment-781725032


   @everglory99 can you please post the YAML file you're using?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org