You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/08 10:08:53 UTC

[GitHub] [airflow] fboissier opened a new issue #21421: Unable to use EKSPodOperator with aws_conn_id parameter

fboissier opened a new issue #21421:
URL: https://github.com/apache/airflow/issues/21421


   ### Apache Airflow Provider(s)
   
   amazon
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==2.4.0
   
   ### Apache Airflow version
   
   2.2.2
   
   ### Operating System
   
   AmazonLinux
   
   ### Deployment
   
   MWAA
   
   ### Deployment details
   
   Out of box deployment of MWAA with Airflow 2.2.2
   
   ### What happened
   
   I try to launch a K8S pod using [EKSPodOperator](https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.4.0/_api/airflow/providers/amazon/aws/operators/eks/index.html#airflow.providers.amazon.aws.operators.eks.EKSPodOperator) and the parameter **aws_conn_id** in order to authenticate to the EKS cluster through IAM / OIDC provider.
   
   The pod is not starting and i've got the following error in my Airflow task log : 
   ```
   [2022-02-08, 10:55:30 CET] {{refresh_config.py:71}} ERROR - exec: process returned 1. Traceback (most recent call last):
     File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
       mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
     File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
       __import__(pkg_name)
     File "/usr/local/lib/python3.7/site-packages/airflow/__init__.py", line 34, in <module>
       from airflow import settings
     File "/usr/local/lib/python3.7/site-packages/airflow/settings.py", line 35, in <module>
       from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
     File "/usr/local/lib/python3.7/site-packages/airflow/configuration.py", line 1129, in <module>
       conf.validate()
     File "/usr/local/lib/python3.7/site-packages/airflow/configuration.py", line 226, in validate
       self._validate_enums()
     File "/usr/local/lib/python3.7/site-packages/airflow/configuration.py", line 253, in _validate_enums
       + f"{value!r}. Possible values: {', '.join(enum_options)}."
   airflow.exceptions.AirflowConfigException: `[logging] logging_level` should not be 'fatal'. Possible values: CRITICAL, FATAL, ERROR, WARN, WARNING, INFO, DEBUG.
   ```
   
   Indeed, the [EKSHook](https://github.com/apache/airflow/blob/providers-amazon/2.4.0/airflow/providers/amazon/aws/hooks/eks.py#L596) is setting **AIRFLOW__LOGGING__LOGGING_LEVEL** to **fatal** and [airflow core](https://github.com/apache/airflow/blob/2.2.2/airflow/configuration.py#L198) is checking that logging level is **FATAL**.
   
   It seems we've got a case sensitive problem.
   
   ### What you expected to happen
   
   I expected my POD starting using the IAM / OIDC provider auth on EKS.
   
   ### How to reproduce
   
   ```python
   start_pod = EKSPodOperator(
           task_id="run_pod",
           cluster_name="<your_eks_cluster_name>",
           pod_name="run_pod",
           image="amazon/aws-cli:latest",
           cmds=["sh", "-c", "ls"],
           labels={"demo": "hello_world"},
           get_logs=True,
           # Delete the pod when it reaches its final state, or the execution is interrupted.
           is_delete_operator_pod=True,
           aws_conn_id="<your_aws_airflow_conn_id>"
       )
   ```
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #21421: Unable to use EKSPodOperator with aws_conn_id parameter

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21421:
URL: https://github.com/apache/airflow/issues/21421#issuecomment-1032431705


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk closed issue #21421: Unable to use EKSPodOperator with aws_conn_id parameter

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21421:
URL: https://github.com/apache/airflow/issues/21421


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org