You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@linkis.apache.org by "CharlieYan24 (via GitHub)" <gi...@apache.org> on 2023/03/29 13:30:57 UTC

[GitHub] [linkis] CharlieYan24 opened a new pull request, #4424: Dev 1.4.0 storge support oss filesystem(#4185)

CharlieYan24 opened a new pull request, #4424:
URL: https://github.com/apache/linkis/pull/4424

### What is the purpose of the change

1, linkis-storge support oss filesystem
2, support store launch log log in OSS bucket, and support fetch log from OSS in job history.
3, support store hive resultSet in OSS bucket, and support view log in job history.

### Related issues/PRs

Related issues: #4185
Related pr:#4185

### Brief change log

- add OSSFileSystem class which extends from FileSystem class, in order to operate file in OSS .
- add BuildOSSSystem class to build OSSFileSystem.
- modify FsRestfulApi class to fix the error when read log and resultSet from OSS.
- modify FsPath class.

to store log and resultSet in OSS:
- add the following configs in conf/linkis-cg-entrance.properties.
eg:
wds.linkis.resultSet.store.path=oss://linkis/tmp/
wds.linkis.filesystem.hdfs.root.path=oss://linkis/tmp/

- add the following configs in engine engineconn plugins conf.
eg:
let me use hive conf for example
modify linkis/lib/linkis-engineconn-plugins/hive/dist/v3.1.3/conf/linkis-engineconn.properties
add the following configs
wds.linkis.fs.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com
wds.linkis.fs.oss.bucket.name=oss://linkis
wds.linkis.fs.oss.accessKeyId=your accessKeyId
wds.linkis.fs.oss.accessKeySecret=your accessKeySecret

### Checklist

- [x] I have read the [Contributing Guidelines on pull requests](https://github.com/facebook/docusaurus/blob/main/CONTRIBUTING.md#pull-requests).
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not, please discuss on the [Linkis mailing list](https://linkis.apache.org/community/how-to-subscribe) first)
- [x] **If this is a code change**: I have written unit tests to fully verify the new behavior.

<!--

Note

1. Mark the PR title as `[WIP] title` until it's ready to be reviewed.
如果PR还未准备好被review，请在标题上添加[WIP]标识(WIP work in progress)

2. Always add/update tests for any changes unless you have a good reason.
除非您有充分的理由，否则任何修改都需要添加/更新测试

3. Always update the documentation to reflect the changes made in the PR.
始终更新文档以反映 PR 中所做的更改

4. After the PR is submitted, please pay attention to the execution result of git action check.
If there is any failure, please adjust it in time
PR提交后，请关注git action check 执行结果，关键的check失败时，请及时修正

5. Before the pr is merged, if the commit is missing, you can continue to commit the code
在未合并前，如果提交有遗漏，您可以继续提交代码

6. After you submit PR, you can add assistant WeChat, the WeChat QR code is
https://user-images.githubusercontent.com/7869972/176336986-d6b9be8f-d1d3-45f1-aa45-8e6adf5dd244.png
您提交pr后，可以添加助手微信，微信二维码为
https://user-images.githubusercontent.com/7869972/176336986-d6b9be8f-d1d3-45f1-aa45-8e6adf5dd244.png

-->

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org

[GitHub] [linkis] peacewong merged pull request #4424: feat : linkis-storage support OSS filesystem support write/read launch log and resultSet in OSS(#4185)

Posted by "peacewong (via GitHub)" <gi...@apache.org>.

peacewong merged PR #4424:
URL: https://github.com/apache/linkis/pull/4424


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org

[GitHub] [linkis] peacewong commented on a diff in pull request #4424: feat : linkis-storage support OSS filesystem support write/read launch log and resultSet in OSS(#4185)

Posted by "peacewong (via GitHub)" <gi...@apache.org>.

peacewong commented on code in PR #4424:
URL: https://github.com/apache/linkis/pull/4424#discussion_r1154116120


##########
linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/io/FsPath.java:
##########
@@ -273,6 +271,10 @@ public String getSchemaPath() {
     if (WINDOWS && !"hdfs".equals(getFsType())) {
       return getFsType() + "://" + uri.getAuthority() + uri.getPath();
     }
+
+    if (uri.getPath().startsWith("/")) {
+      return getFsType() + "://" + uri.getPath().substring(1);

Review Comment:
   The path of hdfs should be hdfs:///tmp



##########
linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/io/FsPath.java:
##########
@@ -169,8 +167,8 @@ public File toFile() {
     return new File(uri);
   }
 
-  public Path toPath() {
-    return FileSystems.getDefault().getPath(uri.toString());
+  public String getUriString() {

Review Comment:
   A modification of this method requires the addition of a single test



##########
tool/dependencies/known-dependencies.txt:
##########
@@ -705,4 +705,16 @@ kerb-simplekdc-1.0.1.jar
 kerb-util-1.0.1.jar
 kerby-asn1-1.0.1.jar
 kerby-config-1.0.1.jar
-kerby-pkix-1.0.1.jar
\ No newline at end of file
+kerby-pkix-1.0.1.jar
+hadoop-aliyun-3.3.4.jar
+aliyun-sdk-oss-3.16.0.jar
+aliyun-java-sdk-core-4.5.10.jar

Review Comment:
   need to add the license to dir: linkis-dist/release-docs/licenses, And the protocol needs to be compatible with apache v2



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org