You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "sjgllgh (via GitHub)" <gi...@apache.org> on 2023/04/03 16:52:42 UTC
[GitHub] [linkis] sjgllgh opened a new pull request, #4434: feat : linkis-storage support S3 filesystem support write/read launch log and resultSet in S3(#4185)
sjgllgh opened a new pull request, #4434:
URL: https://github.com/apache/linkis/pull/4434
### What is the purpose of the change
1. linkis-storge support s3 filesystem;
2. support store/read launch log in s3 bucket;
3. support store/read resultSet in s3 bucket.
### Related issues/PRs
Related issues: #4185
Related pr:#4185
### Brief change log
- add S3FileSystem class which extends from FileSystem class, in order to operate file in S3 .
- add BuildS3System class to build S3FileSystem.
- modify StorageUtils class to fix an error when reading a resultSet from s3 in S3ObjectInputStream.
- modify StorageConfiguration class.
### How to use s3 file system
1. modify configs in conf/linkis-cg-entrance.properties.
- wds.linkis.resultSet.store.path=s3://linkis/tmp/
- wds.linkis.filesystem.hdfs.root.path=s3://linkis/tmp/
2. add configs in conf/linkis.properties.
- wds.linkis.storage.s3.access.key=kyrO2anrUJQLtS2B
- wds.linkis.storage.s3.secret.key=3BO2O9YHW110A64lx4hqh0MNsGHfIpZP
- wds.linkis.storage.s3.endpoint=http://localhost:9000/
- wds.linkis.storage.s3.region=cn-hb-1
- wds.linkis.storage.s3.bucket=bucket-1
### Checklist
- [x] I have read the [Contributing Guidelines on pull requests](https://github.com/facebook/docusaurus/blob/main/CONTRIBUTING.md#pull-requests).
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not, please discuss on the [Linkis mailing list](https://linkis.apache.org/community/how-to-subscribe) first)
- [x] **If this is a code change**: I have written unit tests to fully verify the new behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] sjgllgh closed pull request #4434: feat : linkis-storage support S3 filesystem support write/read launch log and resultSet in S3(#4185)
Posted by "sjgllgh (via GitHub)" <gi...@apache.org>.
sjgllgh closed pull request #4434: feat : linkis-storage support S3 filesystem support write/read launch log and resultSet in S3(#4185)
URL: https://github.com/apache/linkis/pull/4434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org