You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/11 16:42:35 UTC
[GitHub] [airflow] izhangzhihao opened a new issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
izhangzhihao opened a new issue #15319:
URL: https://github.com/apache/airflow/issues/15319
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
Support MinIO & Aliyun OSS & other storage
<!-- A short description of your feature -->
**Use case / motivation**
<!-- What do you want to happen?
Rather than telling us how you might implement this solution, try to take a
step back and describe what you are trying to achieve.
-->
**Are you willing to submit a PR?**
<!--- We accept contributions! -->
**Related Issues**
<!-- Is there currently another issue associated with this? -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
blcksrx commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-817346233
I would like to make PR for creating MinIO Hook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
blcksrx commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-817979477
@izhangzhihao Currently Its possible to use S3 compatible object storage. you have to create an S3 connection in airflow with these extra args:
```
{
"aws_access_key_id":"your_minio_access_key",
"aws_secret_access_key": "your_minio_secret_key",
"host": "http://127.0.0.1:9000"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] coder2j edited a comment on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
coder2j edited a comment on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-953637106
> > @izhangzhihao Currently Its possible to use S3 compatible object storage. you have to create an S3 connection in airflow with these extra args:
> > ```
> > {
> > "aws_access_key_id":"your_minio_access_key",
> > "aws_secret_access_key": "your_minio_secret_key",
> > "host": "http://127.0.0.1:9000"
> > }
> > ```
>
> Yup, this works for OSS:
>
> ```json
> {
> "region_name": "oss-cn-shanghai",
> "host": "https://airflow-logging.oss-cn-shanghai-internal.aliyuncs.com",
> "aws_access_key_id":"ak",
> "aws_secret_access_key": "sk"
> }
> ```
>
> and envs:
>
> ```
> - name: AIRFLOW__LOGGING__REMOTE_LOGGING
> value: "True"
> - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
> value: "s3://airflow-logging"
> - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
> value: "airflow-logging"
> ```
I tried to use MinIO as a local S3 for airflow logging, it didn't work for me using the connection setup suggested (Tested on Mac OS):
`
{
"aws_access_key_id":"your_minio_access_key",
"aws_secret_access_key": "your_minio_secret_key",
"host": "http://127.0.0.1:9000"
}
`
Instead, try to get your docker gateway by the command `sudo docker network inspect bridge` and replace the host IP address with the getaway you get. In my case, it is `172.17.0.1`, so updating the s3 connection as follows will work.
`{
"aws_access_key_id":"your_minio_access_key",
"aws_secret_access_key": "your_minio_secret_key",
"host": "http://172.17.0.1:9000"
} `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] coder2j edited a comment on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
coder2j edited a comment on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-953637106
> > @izhangzhihao Currently Its possible to use S3 compatible object storage. you have to create an S3 connection in airflow with these extra args:
> > ```
> > {
> > "aws_access_key_id":"your_minio_access_key",
> > "aws_secret_access_key": "your_minio_secret_key",
> > "host": "http://127.0.0.1:9000"
> > }
> > ```
>
> Yup, this works for OSS:
>
> ```json
> {
> "region_name": "oss-cn-shanghai",
> "host": "https://airflow-logging.oss-cn-shanghai-internal.aliyuncs.com",
> "aws_access_key_id":"ak",
> "aws_secret_access_key": "sk"
> }
> ```
>
> and envs:
>
> ```
> - name: AIRFLOW__LOGGING__REMOTE_LOGGING
> value: "True"
> - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
> value: "s3://airflow-logging"
> - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
> value: "airflow-logging"
> ```
I tried to use MinIO as a local S3 for airflow logging, it didn't work for me using the connection setup suggested (Tested on Mac OS):
`{
"aws_access_key_id":"your_minio_access_key",
"aws_secret_access_key": "your_minio_secret_key",
"host": "http://127.0.0.1:9000"
} `
Instead, try to get your docker gateway by the command `sudo docker network inspect bridge` and replace the host IP address with the getaway you get. In my case, it is `172.17.0.1`, so updating the s3 connection as follows will work.
`{
"aws_access_key_id":"your_minio_access_key",
"aws_secret_access_key": "your_minio_secret_key",
"host": "http://172.17.0.1:9000"
} `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
blcksrx commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-817967729
It's great! cause MinIO is S3 compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-817336185
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KeynesLee commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
KeynesLee commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-871520529
Will the Minio connection support HTTPS ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] izhangzhihao commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
izhangzhihao commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-817477506
Or we could make `endpoint_url` of the s3 client configurable because our team is using the boto3 client to access Aliyun OSS.
```
def __init_s3_client__(self):
session = boto3.session.Session(
aws_access_key_id=os.environ['OSS_KEY_ID'],
aws_secret_access_key=os.environ['OSS_KEY_SECRET'],
region_name=os.environ['REGION'],
)
s3config = Config(
s3={
'addressing_style': 'virtual'
}
)
return session.client('s3', endpoint_url='https://' + os.environ['OSS_ENDPOINT_URL'], config=s3config)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] izhangzhihao commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
izhangzhihao commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-821189306
> @izhangzhihao Currently Its possible to use S3 compatible object storage. you have to create an S3 connection in airflow with these extra args:
>
> ```
> {
> "aws_access_key_id":"your_minio_access_key",
> "aws_secret_access_key": "your_minio_secret_key",
> "host": "http://127.0.0.1:9000"
> }
> ```
Yup, this works for OSS:
```json
{
"region_name": "oss-cn-shanghai",
"host": "https://airflow-logging.oss-cn-shanghai-internal.aliyuncs.com",
"aws_access_key_id":"ak",
"aws_secret_access_key": "sk"
}
```
and envs:
```
- name: AIRFLOW__LOGGING__REMOTE_LOGGING
value: "True"
- name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
value: "s3://airflow-logging"
- name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
value: "airflow-logging"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] coder2j commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
coder2j commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-953637106
> > @izhangzhihao Currently Its possible to use S3 compatible object storage. you have to create an S3 connection in airflow with these extra args:
> > ```
> > {
> > "aws_access_key_id":"your_minio_access_key",
> > "aws_secret_access_key": "your_minio_secret_key",
> > "host": "http://127.0.0.1:9000"
> > }
> > ```
>
> Yup, this works for OSS:
>
> ```json
> {
> "region_name": "oss-cn-shanghai",
> "host": "https://airflow-logging.oss-cn-shanghai-internal.aliyuncs.com",
> "aws_access_key_id":"ak",
> "aws_secret_access_key": "sk"
> }
> ```
>
> and envs:
>
> ```
> - name: AIRFLOW__LOGGING__REMOTE_LOGGING
> value: "True"
> - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
> value: "s3://airflow-logging"
> - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
> value: "airflow-logging"
> ```
For anyone who is trying to use MinIO as a local S3 for airflow logging, it didn't work for me using the connection setup suggested:
`{
"aws_access_key_id":"your_minio_access_key",
"aws_secret_access_key": "your_minio_secret_key",
"host": "http://127.0.0.1:9000"
} `
Instead, try to get your docker gateway by the command `sudo docker network inspect bridge` and replace the host IP address with the getaway you get. In my case, it is `172.17.0.1`, so updating the s3 connection as followed will work.
`{
"aws_access_key_id":"your_minio_access_key",
"aws_secret_access_key": "your_minio_secret_key",
"host": "http://172.17.0.1:9000"
} `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-817981530
Yes. We can use minio to tests logging for S3-compatibile storage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #15319: [Remote Logging] Support MinIO & Aliyun OSS & other storage
Posted by GitBox <gi...@apache.org>.
blcksrx commented on issue #15319:
URL: https://github.com/apache/airflow/issues/15319#issuecomment-817980468
@mik-laj Shall we provide tests for MinIO & Aliyun OSS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org