You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/02/24 02:44:28 UTC
[GitHub] [incubator-dolphinscheduler] yaofengchn opened a new issue #4858: use s3 resource got error
yaofengchn opened a new issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858
When using S3, i can't create tenants or files, use hdfs is ok
The configuration is as follows. How to modify it
api-server.log
当使用s3的时候，创建租户和创建文件都不行，换成hdfs就可以了
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yaofengchn closed issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
yaofengchn closed issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858#issuecomment-784746865
In 1.3.x version, the configuration in practice is like as follows:
```
resource.storage.type=S3
resource.upload.path=/dolphinscheduler
fs.defaultFS=s3a://dolphinscheduler
fs.s3a.endpoint=http://xxxxxx
fs.s3a.access.key=xxxxxx
fs.s3a.secret.key=xxxxxx
```
As for your configuration, `defautFS` should be updated to `s3a://dolphinscheduler`, and make sure the directory `/dolphinscheduler` in s3 has been created
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858#issuecomment-785020099
@yaofengchn Sorry, I made a mistake. You are right, `dolphinscheduler` in `s3a://dolphinscheduler` stands for bucket. So make sure the bucket `dolphinscheduler2` and the directory `/dolphinscheduler` in s3 has been created, and the tenant associated with the current user has permission to read/write the the bucket `dolphinscheduler2` and the directory `/dolphinscheduler`.
- Check whether the current user is associated with a tenant?
- Check the tenant's read/write permission for the bucket `dolphinscheduler2` in s3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen commented on issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
chengshiwen commented on issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858#issuecomment-785020099
@yaofengchn Sorry, I made a mistake. You are right, `dolphinscheduler` in `s3a://dolphinscheduler` stands for bucket. So make sure the bucket `dolphinscheduler2` and the directory `/dolphinscheduler` in s3 has been created, and the tenant associated with the current user has permission to read/write the the bucket `dolphinscheduler2` and the directory `/dolphinscheduler`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yaofengchn commented on issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
yaofengchn commented on issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858#issuecomment-785004549
must be s3a://dolphinscheduler ？
fs.defaultFS = s3a://dolphinscheduler
can not use s3a://dolphinscheduler2 even if i had create the bucket s3a://dolphinscheduler2?
必须是s3a://dolphinscheduler这个桶吗？
我现在是创建了s3a://dolphinscheduler2 桶，然后使用这个，报了上面的错
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen commented on issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
chengshiwen commented on issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858#issuecomment-784746865
In 1.3.x version, the configuration is like as follows:
```
resource.storage.type=S3
fs.defaultFS=s3a://dolphinscheduler
fs.s3a.endpoint=http://xxxxxx
fs.s3a.access.key=xxxxxx
fs.s3a.secret.key=xxxxxx
```
As for your configuration, maybe `defautFS` should be updated to `s3a://dolphinscheduler`, and make sure the directory `/dolphinscheduler` in s3 has been created
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yaofengchn commented on issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
yaofengchn commented on issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858#issuecomment-785788380
@chengshiwen
thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858#issuecomment-784746865
In 1.3.x version, the configuration is like as follows:
```
resource.storage.type=S3
resource.upload.path=/dolphinscheduler
fs.defaultFS=s3a://dolphinscheduler
fs.s3a.endpoint=http://xxxxxx
fs.s3a.access.key=xxxxxx
fs.s3a.secret.key=xxxxxx
```
As for your configuration, `defautFS` should be updated to `s3a://dolphinscheduler`, and make sure the directory `/dolphinscheduler` in s3 has been created
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #4858: use s3 resource got error
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #4858:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4858#issuecomment-784746865
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org