You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/06/07 10:57:32 UTC
[GitHub] [dolphinscheduler] geosmart opened a new issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
geosmart opened a new issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599
I need to change hdfs to minio , but the endpoint config `fs.s3a.endpoint` not working
## test case
```java
public static void main(String[] args) throws Exception {
test_minio_s3a();
}
private static void test_minio_s3a() throws IOException {
Configuration configuration = new HdfsConfiguration();
configuration.set("fs.defaultFS", "s3a://bucketname");
configuration.set("fs.s3a.endpoint", "http://ip:32030");
configuration.set("fs.s3a.access.key", "ak");
configuration.set("fs.s3a.secret.key", "sk");
configuration.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
FileSystem fs = FileSystem.get(configuration);
System.out.println(fs.listFiles(new Path("/"), false));
}
```
## error log
```java
8:50:55.856 [main] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection released: [id: 0][route: {s}->https://bucketname.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 15; total allocated: 0 of 15]
18:50:55.861 [main] DEBUG com.amazonaws.request - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: null, AWS Request ID: X41SEJ67XC25KHPD, AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: KKy0QdemQ+TggcpuXXQKgg8R+Yac2BRgBO6Vs2Ed524epxg2RZnxy6ObR4Wpq0xMscXcJlb/Mec=
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: X41SEJ67XC25KHPD, AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: KKy0QdemQ+TggcpuXXQKgg8R+Yac2BRgBO6Vs2Ed524epxg2RZnxy6ObR4Wpq0xMscXcJlb/Mec=
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:154)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at org.apache.dolphinscheduler.common.dboard.HadoopUtilsTest.test_minio_s3a(HadoopUtilsTest.java:35)
at org.apache.dolphinscheduler.common.dboard.HadoopUtilsTest.main(HadoopUtilsTest.java:24)
Disconnected from the target VM, address: '127.0.0.1:37581', transport: 'socket'
```
Any help would be appreciated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-858637563
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart closed issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
geosmart closed issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-855828805
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart edited a comment on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
geosmart edited a comment on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-855860072
resolved
hadoop-aws `2.6.0` not support set endpoint outside, it use the default `s3.amazonaws.com` by
setEndpoint(Constants.S3_HOSTNAME);
and in `2.7.2`
```
this.s3 = new AmazonS3Client(credentials, awsConf);
proxyUsername = conf.getTrimmed("fs.s3a.endpoint", "");
if (!proxyUsername.isEmpty()) {
try {
this.s3.setEndpoint(proxyUsername);
} catch (IllegalArgumentException var23) {
msg = "Incorrect endpoint: " + var23.getMessage();
LOG.error(msg);
throw new IllegalArgumentException(msg, var23);
}
}
```
we can overwrite the endpoint with `fs.s3a.endpoint` config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
geosmart commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-855861454
related pr [267-support to store resources in s3a(support aws s3 & minio) ](https://github.com/apache/dolphinscheduler/pull/267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
geosmart commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-858616002
> @geosmart 你好,问一下s3怎么配才可以呢,data.store2hdfs.basepath 这个才是桶中路径? 不是resource.upload.path 这个吗?
> 我现在配置是
> FS_DEFAULT_FS: s3a://桶名称
> FS_S3A_ACCESS_KEY: xxxxx
> FS_S3A_ENDPOINT: http://s3.xxx.com
> FS_S3A_SECRET_KEY: xxxxx
> RESOURCE_STORAGE_TYPE: S3
> RESOURCE_UPLOAD_PATH: /dolphinscheduler
> 但是创建租户会失败.
`data.store2hdfs.basepath ` is not working any more
`resource.upload.path` is the bucket root path
bucktname is config in `fs.defaultFS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] nmgliangwei commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
nmgliangwei commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-857383543
@geosmart 你好,问一下s3怎么配才可以呢,data.store2hdfs.basepath 这个才是桶中路径? 不是resource.upload.path 这个吗?
我现在配置是
FS_DEFAULT_FS: s3a://桶名称
FS_S3A_ACCESS_KEY: xxxxx
FS_S3A_ENDPOINT: http://s3.xxx.com
FS_S3A_SECRET_KEY: xxxxx
RESOURCE_STORAGE_TYPE: S3
RESOURCE_UPLOAD_PATH: /dolphinscheduler
但是创建租户会失败.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
geosmart commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-855860072
resolved
hadoop-aws 2.6.0 not support set endpoint outside, it use the default "s3.amazonaws.com" by
setEndpoint(Constants.S3_HOSTNAME);
and in 2.7.2
```
this.s3 = new AmazonS3Client(credentials, awsConf);
proxyUsername = conf.getTrimmed("fs.s3a.endpoint", "");
if (!proxyUsername.isEmpty()) {
try {
this.s3.setEndpoint(proxyUsername);
} catch (IllegalArgumentException var23) {
msg = "Incorrect endpoint: " + var23.getMessage();
LOG.error(msg);
throw new IllegalArgumentException(msg, var23);
}
}
```
we can overwrite the endpoint with `fs.s3a.endpoint` config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-858636341
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
geosmart commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-857346460
if you want to migration hdfs to minio
don't remember to copy the hdfs data of `resource.upload.path` to minio(the same path)
trick: you can mount both hdfs and minio with `fuse` ,and do copy in local
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
geosmart commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-856397577
I found that 'data.store2hdfs.basepath' config param not exist any usage.
and the HadoopUtils is init root path by `resource.upload.path`
the common.propertiesof resource config for minio server
---
```ini
# resource config of minio server
resource.storage.type=S3
resource.upload.path=/dolphinscheduler
# minio bucket name
fs.defaultFS=s3a://dolphinscheduler
fs.s3a.endpoint=http://minioserver_ip:minioserver_port
fs.s3a.access.key=ak
fs.s3a.secret.key=sk
# root path in bucket
data.store2hdfs.basepath=/dolphinscheduler
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart commented on issue #5599: [Resource] fs.s3a.endpoint not working when change hdfs to minio
Posted by GitBox <gi...@apache.org>.
geosmart commented on issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599#issuecomment-858615145
> @geosmart 你好,问一下s3怎么配才可以呢,data.store2hdfs.basepath 这个才是桶中路径? 不是resource.upload.path 这个吗?
> 我现在配置是
> FS_DEFAULT_FS: s3a://桶名称
> FS_S3A_ACCESS_KEY: xxxxx
> FS_S3A_ENDPOINT: http://s3.xxx.com
> FS_S3A_SECRET_KEY: xxxxx
> RESOURCE_STORAGE_TYPE: S3
> RESOURCE_UPLOAD_PATH: /dolphinscheduler
> 但是创建租户会失败.
s3 now does not support in ds ,the configiguration only support s3a
if you want to use s3, you should change the config like this
```
fs.s3.awsAccessKeyId=xxx
fs.s3.awsSecretAccessKey=xxx
```
and do some impl in the HadoopUtils class
ref [hadoop-aws](http://hadoop.apache.org/docs/r2.10.1/hadoop-aws/tools/hadoop-aws/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org