You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hanleybrand (via GitHub)" <gi...@apache.org> on 2025/08/20 09:10:47 UTC
Re: [D] Issue troubleshooting an s3 connection for remote logging to Min.io [airflow]
GitHub user hanleybrand closed a discussion: Issue troubleshooting an s3 connection for remote logging to Min.io
I'm trying to configure a Kubernetes deployment of Airflow 2.4.1 to use remote logging, but I can't create a working Amazon Web Services connection in the Airflow UI (or at least I get an error when I test the connection)
### TLDR;
While searching and hacking around seem to have solved some of the issue (or at least fixed some other issues I hadn't been aware of?) when I use either the UI or python route to test_connection I get an error, but every other way I connect seems to work.
The error I get when testing the connection is:
> **An error occurred (InvalidParameterValue) when calling the GetCallerIdentity operation: Unsupported action GetCallerIdentity**
The biggest issue of course, is that when I deploy/redeploy to the kubernetes cluster logs are not being written to the 3s bucket, which makes it difficult to check the logs to see what's going on 😉
My guess at the problem when I look at the code ( [AwsGenericHook](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/hooks/base_aws.py#L631)? ) that I think is generating that error, the test_connection function is checking the connection using STS (session tags), which I don't have configured (I don't know if min.io supports STS actually, it looks like it supports AssumeRoleWithWebIdentity )
### My configuration:
Anonymizing my config particulars, let's say Airflow is deployed to `https://airflow.k8s.example.com`
In Admin -> Connections I have a conn_id 'amazon_s3' of type 'Amazon Web Services' with the correct AWS Access Key ID & Secret Access Keys set, plus Extra set to:
```
{ "aws_access_key_id": "LAWGIN",
"aws_secret_access_key": "PASSWOID",
"endpoint_url": "https://minio01.k8s.example.com:9000",
"region_name": "us-east-1"
"encrypted": "yes", "verify": "False"}
```
**Aiflow.cfg logging section**
```cfg
[logging]
colored_console_log = False
encrypt_s3_logs = False
remote_base_log_folder = s3://airflow-logs-dev
remote_log_conn_id = amazon_s3
remote_logging = True
```
**Notes**:
- "encrypted" set to "no" or "yes" does not change what error I got.
- "verify" can likewise be set to "False", "None" or "True" -- the only thing that changes the error is removing the key/value pair for verify
- I can also remove the login/password k/vs from the Exrta JSON string, it doesn't seem to matter. I leave them in because some of the discussions implied that they needed to be in the extra string.
With "verify" key included I get the following error:
> **- SSL validation failed for https://minio01.k8s.example.com:9000/ [Errno 2] No such file or directory**
If I remove the "verify" key entirely, I also get this error.
> **An error occurred (InvalidParameterValue) when calling the GetCallerIdentity operation: Unsupported action GetCallerIdentity**
### HIstory, stuff I've tried, etc
I had started with what the docs say, which was the simpler extra set to
{ "aws_access_key_id": "LAWGIN", "aws_secret_access_key": "PASSWOID",
"host": "https://minio01.k8s.example.com:9000"} but I got errors regarding the region not being set correctly.
I have confirmed with the aws-cli and min.io mc clients that the aws_access_key_id, aws_secret_access_key and endpoint_url paremters above are correct, e.g.:
```shell
aws --endpoint-url https://minio01.k8s.example.com:9000 s3 ls s3://airflow-logs-dev
```
and
``` shell
mc alias set aa https://minio01.k8s.example.com:9000 LAWGIN PASSWOID
mc ls aa/airflow-logs-dev
```
both allow me to interact with the Min.io instance, so I am certain the issue is not a typo in my login/password, etc. (I copy/pasted values out of my config to test with aws/mc)
Additionally, I can execute a shell in the webserver pod and using python in the shell:
```python
>>> import boto3
>>> client = boto3.client(
... 's3',
... aws_access_key_id='LAWGIN',
... aws_secret_access_key='PASSWOID',
... region_name='us-east-1',
... endpoint_url='https://minio01.k8s.example.com:9000',
... )
>>>
>>> client.list_buckets()
{'ResponseMetadata': {'RequestId': '171CD39BF1DA4102', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'accept-ranges': 'bytes', 'content-length': '461', 'content-security-policy': 'block-all-mixed-content', 'content-type': 'application/xml', 'server': 'MinIO', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'vary': 'Origin, Accept-Encoding', 'x-amz-bucket-region': 'us-east-1', 'x-amz-request-id': '171CD39BF1DA4102', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'date': 'Mon, 10 Oct 2022 21:50:28 GMT'}, 'RetryAttempts': 0}, 'Buckets': [{'Name': 'airflow-logs-dev', 'CreationDate': datetime.datetime(2022, 10, 3, 16, 26, 7, 938000, tzinfo=tzlocal())},....#etc
```
There was a clue that it it might be certificate related so I tried adding `RUN cp $(python -m certifi) /etc/ssl/certs/` to my Dockerfile, and while it seems to help (no longer get a cert error with `S3Hook.check_for_bucket()`) I'm back to getting the InvalidParameterValue/GetCallerIdentity error
```python
>>> from airflow.providers.amazon.aws.hooks.s3 import S3Hook
>>> s3h = S3Hook('amazon_s3')
>>> b = 'airflow-logs-dev'
>>> s3h.check_for_bucket(b)
[2022-10-11T02:39:41.134+0000] {base.py:71} INFO - Using connection ID 'amazon_s3' for task execution.
[2022-10-11T02:39:41.137+0000] {connection_wrapper.py:303} INFO - AWS Connection (conn_id='amazon_s3', conn_type='aws') credentials retrieved from login and password.
True
# that's good
>>> s3h.test_connection()
(False, 'An error occurred (InvalidParameterValue) when calling the GetCallerIdentity operation: Unsupported action GetCallerIdentity')
# not as good
```
Is anyone familiar with setting this kind of connection/logging config up on the more recent versions of Airflow? As far as I can tell the error is coming from the [AwsGenericHook](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/hooks/base_aws.py#L631) class, but it seems like an error with session tokens (STS) which isn't what I'm using in my config, so it's confusing.
edit: edited for clarity, hopefully
GitHub link: https://github.com/apache/airflow/discussions/26979
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org