[GitHub] [airflow] barrywhart opened a new issue, #25827: KubernetesPodOperator "env_vars" field is documented to be templated, but it doesn't work

[GitBox <gi...@apache.org>]

barrywhart opened a new issue, #25827:
URL: https://github.com/apache/airflow/issues/25827

   ### Apache Airflow version
   
   main (development)
   
   ### What happened
   
   The KubernetesPodOperator `env_vars` field is [documented to be templated](https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L99), but it doesn't work. If I pass a JSON string to the constructor, it throws an exception:
   ```
     File "/opt/airflow/airflow/models/baseoperator.py", line 410, in apply_defaults
       result = func(self, **kwargs, default_args=default_args)
     File "/opt/airflow/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 236, in __init__
       self.env_vars = convert_env_vars(env_vars) if env_vars else []
     File "/opt/airflow/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py", line 105, in convert_env_vars
       raise AirflowException(f"Expected dict or list, got {type(env_vars)}")
   airflow.exceptions.AirflowException: Expected dict or list, got <class 'str'>
   ```
   
   ### What you think should happen instead
   
   If it receives a string, the constructor should convert it to Python (probably using `ast.literal_eval()`).
   
   ### How to reproduce
   
   I've written a test to demonstrate the behavior. It could be added to `kubernetes_tests/test_kubernetes_pod_operator.py`.
   ```
       def test_env_vars_are_templatized(self):
           # WHEN
           env_vars = """
               [
                   {
                       "name": "ENV1",
                       "value": "val1",
                       "value_from": null
                   },
                   {
                       "name": "ENV2",
                       "value": "val2",
                       "value_from": null
                   },
                   {
                       "name": "ENV3",
                       "value": null,
                       "value_from": {
                           "config_map_key_ref": null,
                           "field_ref": {
                               "api_version": null,
                               "field_path": "status.podIP"
                           },
                           "resource_field_ref": null,
                           "secret_key_ref": null
                       }
                   }
               ]
           """
           k = KubernetesPodOperator(
               namespace='default',
               image="ubuntu:16.04",
               cmds=["bash", "-cx"],
               arguments=["echo 10"],
               env_vars=env_vars,
               labels={"foo": "bar"},
               name="test-" + str(random.randint(0, 1000000)),
               task_id="task" + self.get_current_task_name(),
               in_cluster=False,
               do_xcom_push=False,
           )
           # THEN
           context = create_context(k)
           actual_pod = self.api_client.sanitize_for_serialization(k.build_pod_request_obj(context))
           self.expected_pod['spec']['containers'][0]['env'] = [
               {'name': 'ENV1', 'value': 'val1'},
               {'name': 'ENV2', 'value': 'val2'},
               {'name': 'ENV3', 'valueFrom': {'fieldRef': {'fieldPath': 'status.podIP'}}},
           ]
           assert self.expected_pod == actual_pod
   ```
   
   ### Operating System
   
   Debian GNU/Linux 11 (bullseye) (I'm running the Breeze dev environment)
   
   ### Versions of Apache Airflow Providers
   
   I'm using the Breeze development environment with the latest `main` code as of August 19, 2022:
   ```
   commit 0254f30a5a90f0c3104782525fabdcfdc6d3b7df (HEAD -> main, origin/main, origin/HEAD)
   Author: Ash Berlin-Taylor <as...@apache.org>
   Date:   Fri Aug 19 14:15:43 2022 +0100
   
       Let timetables control generated run_ids. (#25795)
   ```
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   Breeze development environment
   
   ### Anything else
   
   The problem happens every time.
   
   I would like to submit a PR to address this issue. I would also like (**if it sounds reasonable -- please let me know**) to add similar templating support for these other fields:
   * `resources`
   * `tolerations`
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #25827: KubernetesPodOperator "env_vars" field is documented to be templated, but it doesn't work

[GitBox <gi...@apache.org>]

potiuk closed issue #25827: KubernetesPodOperator "env_vars" field is documented to be templated, but it doesn't work
URL: https://github.com/apache/airflow/issues/25827


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org