You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/09 16:07:06 UTC
[GitHub] [airflow] FelixKJose opened a new issue #11378: Vault kv_engine_version 1 configuration is not working in Airflow
FelixKJose opened a new issue #11378:
URL: https://github.com/apache/airflow/issues/11378
Hello,
My Vault is on **kv_engine_version is 1**. So I tried to configure that as follows in airflow.cfg:
`backend_kwargs = {"url": "https://vproxy.us-east.**************.com", "kv_engine_version": 1, "auth_type": "approle", "role_id": "d48b422d-f2cb-****-7781-206104fa7367", "secret_id": "ac48613e-*****-8638-125a-dce1e60aa115", "mount_point": "cf/243d677f-dbe3-****-babc-970ce67ebf3b/secret"}`
But when I run a **DAG** which is trying to pull a connection which is stored in vault, I am getting following error:
> [2020-10-08 20:07:18,972] {taskinstance.py:900} INFO - Executing <Task(PythonOperator): vault_python_task> on 2020-10-09T00:02:06.044874+00:00
[2020-10-08 20:07:18,976] {standard_task_runner.py:53} INFO - Started process 34072 to run task
[2020-10-08 20:07:19,068] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: tutorial_v1.7.vault_python_task 2020-10-09T00:02:06.044874+00:00 [running]> MACC02Y42HGJG5J
[2020-10-08 20:07:19,449] {taskinstance.py:1145} ERROR - The conn_id `s3_connect2` isn't defined
> ....
> raise AirflowException("The conn_id `{0}` isn't defined".format(conn_id))
airflow.exceptions.AirflowException: The conn_id `s3_connect2` isn't defined
**Note:** s3_connect2 is present in vault and I am able to get that value via a python script using `hvac library as airflow does internally`
When I remove **kv_engine_version = 1** and has following:
`backend_kwargs = {"url": "https://vproxy.us-east.**************.com", "auth_type": "approle", "role_id": "d48b422d-f2cb-****-7781-206104fa7367", "secret_id": "ac48613e-*****-8638-125a-dce1e60aa115", "mount_point": "cf/243d677f-dbe3-****-babc-970ce67ebf3b/secret"}`
Then I am getting following error:
> [2020-10-09 10:04:37,011] {logging_mixin.py:112} INFO - [2020-10-09 10:04:37,010] {hashicorp_vault.py:196} INFO - Secret s3_connect2 not found in Path: connections/s3_connect2
[2020-10-09 10:04:37,144] {taskinstance.py:1145} ERROR - The conn_id `s3_connect2` isn't defined
Here I can see that its hitting **hashicorp_vault.py** but when I have **kv_engine_version = 1**, its not even hitting **hashicorp_vault.py**. So can someone help me on what am I doing wrong?
**Apache Airflow version**: 1.10.10
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): No, Local machine (MacOS)
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): Mac OS
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
getting error for an existing vault secret key:
> The conn_id `s3_connect2` isn't defined
**What you expected to happen**:
expected to return the connection from the vault for the given connection_id
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #11378: Vault kv_engine_version 1 configuration is not working in Airflow
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #11378:
URL: https://github.com/apache/airflow/issues/11378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org