You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/22 09:46:53 UTC
[GitHub] [airflow] vaddisrinivas opened a new issue #9474: Airflow support for S3 compatible storages
vaddisrinivas opened a new issue #9474:
URL: https://github.com/apache/airflow/issues/9474
Hi,
Curious to know about the support for S3 compatible storages like DELL ECS, MINIO ETC
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimon222 commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
dimon222 commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-647581742
Yes, it works as long as you specify endpoint url. However, it seems to be broken to serving logs in UI for some users on latest release 1.10.10 (works fine in 1.10.9)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Asgoret commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
Asgoret commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-651794943
Everywhere is HTTP, but what about HTTPS? airflow doesn't support HTTPS S3 endpoints? I've got a very odd error (`An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records`) when use HTTPS-based endpoint. In dev with HTTP, all works perfectly.
cc @dimon222
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimon222 commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
dimon222 commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-650151403
> hi @dispensable / all,
>
> How do I pass the access_key_id and other parameters to connect to that? can someone please help on that?
Username/Password fields in respective connection
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dispensable commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
dispensable commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-648721448
We are currently using Minio as our airflow cluster logging backend. It works perfect. But setting the connection is a little bit tricky U should not set host in the host form field otherwise set host in the `Extra` fields with `{"host": "http://YOUR_S3_URL:PORT"}`, just leave the `host`/`port`/`schema` form fields blank.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vaddisrinivas commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
vaddisrinivas commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-650155549
hi @dispensable/@dimon222, thanks for your help.
I have SUCCESSFULLY enabled pushing airflow logs to s3 compatible bucket by following what was mentioned above,
in brief :
- created a new connection with -> {"host": "http://myhost:myport", "aws_access_key_id" :"myaccesskey/username"
,"aws_secret_access_key": "myreallybigsecretkey"}
- then I went ahead, modified the configuration by adding
AIRFLOW__CORE__REMOTE_LOGGING: True,
AIRFLOW__CORE__REMOTE_LOG_CONN_ID: connectionthatwassetearlier,
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://bucketname/pathorfolderonBucket",
- restarted Airflow!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vaddisrinivas closed issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
vaddisrinivas closed issue #9474:
URL: https://github.com/apache/airflow/issues/9474
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vaddisrinivas commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
vaddisrinivas commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-650138910
hi @dispensable / all,
How do I pass the access_key_id and other parameters to connect to that? can someone please help on that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimon222 edited a comment on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
dimon222 edited a comment on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-650151403
> hi @dispensable / all,
>
> How do I pass the access_key_id and other parameters to connect to that? can someone please help on that?
Username/Password fields in respective connection
Extra args I believe not supported apart of mentioned above host.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vaddisrinivas commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
vaddisrinivas commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-647650974
It doesnt work for me even as I continue to provide HOST in the connection and also other relevant parameters.
If there is any alternative for the same, please help with that.
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dispensable edited a comment on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
dispensable edited a comment on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-648721448
We are currently using Ceph RGW as our airflow cluster logging backend. It works perfect. But setting the connection is a little bit tricky U should not set host in the host form field otherwise set host in the `Extra` fields with `{"host": "http://YOUR_S3_URL:PORT"}`, just leave the `host`/`port`/`schema` form fields blank.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vaddisrinivas edited a comment on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
vaddisrinivas edited a comment on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-647650974
It doesnt work for me even as I continue to provide HOST in the connection and also other relevant parameters.
If there is any alternative for the same, please help with that.
Also @dimon222, can you please share a sample connection screenshot/ instruction along with configurations thus enabled to facilitate remote logging?
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vaddisrinivas commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
vaddisrinivas commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-648724411
hi @dispensable , will try this and update the ticket asap!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Asgoret edited a comment on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
Asgoret edited a comment on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-651794943
Everywhere is HTTP, but what about HTTPS? airflow doesn't support HTTPS S3 endpoints? I've got a very odd error (`An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records`) when use HTTPS-based endpoint. In dev with HTTP, all works perfectly.
cc @dimon222
ok, how it works.
Connection in GUI:
```
Name: Some name
Type: S3
Host: <empty>
Schema: <empty>
Login: Your ID from minio
Password: You password key
Port: <empty>
Extras: {"host": "https://domain"}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9474: Airflow support for S3 compatible storages
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-647408995
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org