You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/09/29 13:13:59 UTC
[GitHub] [incubator-doris] gengjun-git opened a new pull request #4690: Broker oss support
gengjun-git opened a new pull request #4690:
URL: https://github.com/apache/incubator-doris/pull/4690
Support aliyun oss read/write in broker, through hadoop-oss client.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #4690: [Feature]Broker oss support
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #4690:
URL: https://github.com/apache/incubator-doris/pull/4690#discussion_r500383881
##########
File path: fs_brokers/apache_hdfs_broker/build.sh
##########
@@ -42,6 +42,12 @@ rm -rf ${BROKER_OUTPUT}
install -d ${BROKER_OUTPUT}/bin ${BROKER_OUTPUT}/conf \
${BROKER_OUTPUT}lib/
+# download aliyun hadoop oss jar, this can not be found in maven repository, so we download it manually
+wget http://gosspublic.alicdn.com/hadoop-spark/hadoop-oss-2.7.2.tar.gz
+tar xzf hadoop-oss-2.7.2.tar.gz
+mv hadoop-oss-2.7.2/hadoop-aliyun-2.7.2.jar ${BROKER_OUTPUT}/lib/
+rm -r hadoop-oss-2.7.2 hadoop-oss-2.7.2.tar.gz
Review comment:
Is it must be 2.7.2?
Does this not work?
`https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aliyun/2.9.2`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gengjun-git commented on a change in pull request #4690: [Feature]Broker oss support
Posted by GitBox <gi...@apache.org>.
gengjun-git commented on a change in pull request #4690:
URL: https://github.com/apache/incubator-doris/pull/4690#discussion_r546200728
##########
File path: docs/zh-CN/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
##########
@@ -477,6 +485,21 @@ under the License.
ORDER BY source_sequence
)
with BROKER "hdfs" ("username"="user", "password"="pass");
+
+ 15. 从 Aliyun OSS 导入csv格式的数据
+ LOAD LABEL example_db.label15
+ (
+ DATA INFILE("oss://my_bucket/input/file.csv")
+ INTO TABLE `my_table`
+ (k1, k2, k3)
+ )
+ WITH BROKER my_oss_broker
+ (
+ "fs.oss.accessKeyId" = "xxxxxxxxxxxxxxxxxxxxxxxxxx",
Review comment:
we use hadoop oss api to access aliyun oss, and the config params is `fs.oss.accessKeyId`, `fs.oss.accessKeySecret`. https://developer.aliyun.com/article/292792
And like s3 config, we'd better use the same param to hadoop ecology.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gengjun-git commented on a change in pull request #4690: [Feature]Broker oss support
Posted by GitBox <gi...@apache.org>.
gengjun-git commented on a change in pull request #4690:
URL: https://github.com/apache/incubator-doris/pull/4690#discussion_r546200728
##########
File path: docs/zh-CN/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
##########
@@ -477,6 +485,21 @@ under the License.
ORDER BY source_sequence
)
with BROKER "hdfs" ("username"="user", "password"="pass");
+
+ 15. 从 Aliyun OSS 导入csv格式的数据
+ LOAD LABEL example_db.label15
+ (
+ DATA INFILE("oss://my_bucket/input/file.csv")
+ INTO TABLE `my_table`
+ (k1, k2, k3)
+ )
+ WITH BROKER my_oss_broker
+ (
+ "fs.oss.accessKeyId" = "xxxxxxxxxxxxxxxxxxxxxxxxxx",
Review comment:
we use hadoop oss api to access aliyun oss, and the config params is fs.oss.accessKeyId, fs.oss.accessKeySecret. https://developer.aliyun.com/article/292792
And like s3 config, we'd better use the same param to hadoop ecology.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng commented on a change in pull request #4690: [Feature]Broker oss support
Posted by GitBox <gi...@apache.org>.
wuyunfeng commented on a change in pull request #4690:
URL: https://github.com/apache/incubator-doris/pull/4690#discussion_r544999570
##########
File path: docs/zh-CN/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
##########
@@ -477,6 +485,21 @@ under the License.
ORDER BY source_sequence
)
with BROKER "hdfs" ("username"="user", "password"="pass");
+
+ 15. 从 Aliyun OSS 导入csv格式的数据
+ LOAD LABEL example_db.label15
+ (
+ DATA INFILE("oss://my_bucket/input/file.csv")
+ INTO TABLE `my_table`
+ (k1, k2, k3)
+ )
+ WITH BROKER my_oss_broker
+ (
+ "fs.oss.accessKeyId" = "xxxxxxxxxxxxxxxxxxxxxxxxxx",
Review comment:
```suggestion
"fs.oss.accessKey" = "xxxxxxxxxxxxxxxxxxxxxxxxxx",
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
##########
@@ -227,8 +227,8 @@ private static void checkPath(String path) throws AnalysisException {
URI uri = new URI(path);
String schema = uri.getScheme();
if (schema == null || (!schema.equalsIgnoreCase("bos") && !schema.equalsIgnoreCase("afs")
- && !schema.equalsIgnoreCase("hdfs"))) {
- throw new AnalysisException("Invalid export path. please use valid 'HDFS://', 'AFS://' or 'BOS://' path.");
+ && !schema.equalsIgnoreCase("hdfs") && !schema.equalsIgnoreCase("oss"))) {
Review comment:
If many `fs` or `object storage`, we can use a `enmu` to judge this
##########
File path: docs/zh-CN/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
##########
@@ -477,6 +485,21 @@ under the License.
ORDER BY source_sequence
)
with BROKER "hdfs" ("username"="user", "password"="pass");
+
+ 15. 从 Aliyun OSS 导入csv格式的数据
+ LOAD LABEL example_db.label15
+ (
+ DATA INFILE("oss://my_bucket/input/file.csv")
+ INTO TABLE `my_table`
+ (k1, k2, k3)
+ )
+ WITH BROKER my_oss_broker
+ (
+ "fs.oss.accessKeyId" = "xxxxxxxxxxxxxxxxxxxxxxxxxx",
+ "fs.oss.accessKeySecret" = "yyyyyyyyyyyyyyyyyyyy",
Review comment:
```suggestion
"fs.oss.secretKey" = "yyyyyyyyyyyyyyyyyyyy",
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gengjun-git commented on pull request #4690: [Feature]Broker oss support
Posted by GitBox <gi...@apache.org>.
gengjun-git commented on pull request #4690:
URL: https://github.com/apache/incubator-doris/pull/4690#issuecomment-706472342
> fe hard-coded to check the schema, oss should added here. and i had checcked it by mine oss broker
> `https://github.com/apache/incubator-doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java`
> 
Ok, I have changed and commited.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gengjun-git commented on a change in pull request #4690: [Feature]Broker oss support
Posted by GitBox <gi...@apache.org>.
gengjun-git commented on a change in pull request #4690:
URL: https://github.com/apache/incubator-doris/pull/4690#discussion_r502734853
##########
File path: fs_brokers/apache_hdfs_broker/build.sh
##########
@@ -42,6 +42,12 @@ rm -rf ${BROKER_OUTPUT}
install -d ${BROKER_OUTPUT}/bin ${BROKER_OUTPUT}/conf \
${BROKER_OUTPUT}lib/
+# download aliyun hadoop oss jar, this can not be found in maven repository, so we download it manually
+wget http://gosspublic.alicdn.com/hadoop-spark/hadoop-oss-2.7.2.tar.gz
+tar xzf hadoop-oss-2.7.2.tar.gz
+mv hadoop-oss-2.7.2/hadoop-aliyun-2.7.2.jar ${BROKER_OUTPUT}/lib/
+rm -r hadoop-oss-2.7.2 hadoop-oss-2.7.2.tar.gz
Review comment:
Not work, I have tried the 2+ and 3+ version of lib from maven repository. So we have to download it by hand.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org