You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/24 09:11:11 UTC
[GitHub] [airflow] dungdm93 opened a new issue #17202: S3 remote logs InvalidObjectName
dungdm93 opened a new issue #17202:
URL: https://github.com/apache/airflow/issues/17202
**Apache Airflow version**: `2.1.2+d25854dd413aa68ea70fb1ade7fe01425f456192`
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): `v1.19.10-gke.1600`
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
My airflow cluster use S3 remote logs to MinIO (a S3 compatible object store) follow [this guide](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/logging/s3-task-handler.html). But when some (example) DAG run, I got the following error:
After some investigation, I find out that airflow don't truncate `s3://<bucket>` part from `remote_base_log_folder` when uploading logs to S3.
(`mc admin trace <target>`)
**What you expected to happen**:
`s3://<bucket>` MUST be truncated from `remote_base_log_folder` or allow config `remote_base_log_folder` without `s3://` prefix and bucket. (only base path)
**How to reproduce it**:
* Airflow config:
* Airflow logs connection:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17202: S3 remote logs InvalidObjectName
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17202:
URL: https://github.com/apache/airflow/issues/17202
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17202: S3 remote logs InvalidObjectName
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17202:
URL: https://github.com/apache/airflow/issues/17202#issuecomment-886093619
If you think you can improve that - feel free to make PR ! It's super easy - there is a "Suggest a change on this page" button when you look at our docs. Actually - people like you are the best to improve the docs, somehow you got the notion that you should get the schema (As a new user) so you are likely the best person to tell how to improve the docs so that others do not feel in the same trap.
Airflow is community managed project and many users contribute back - which is best way to give back to the community for the free software you get. Will you join those users @dungdm93 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dungdm93 commented on issue #17202: S3 remote logs InvalidObjectName
Posted by GitBox <gi...@apache.org>.
dungdm93 commented on issue #17202:
URL: https://github.com/apache/airflow/issues/17202#issuecomment-886028689
I just dig into source code and resolve it myself by remove `Schema` from connection. Magic :metal:
So you guys think we should at least note this one into docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org