You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/02/26 14:35:54 UTC
[GitHub] [incubator-doris] frwrdt opened a new pull request #3004: Support
Amazon S3 data source in Broker Load
frwrdt opened a new pull request #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004
issue #3003
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HangyuanLiu commented on a change in pull request
#3004: Support Amazon S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
HangyuanLiu commented on a change in pull request #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#discussion_r384556201
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -146,12 +182,12 @@ public BrokerFileSystem getFileSystem(String path, Map<String, String> propertie
String password = properties.getOrDefault(PASSWORD_KEY, "");
String dfsNameServices = properties.getOrDefault(DFS_NAMESERVICES_KEY, "");
String authentication = properties.getOrDefault(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
- AUTHENTICATION_SIMPLE);
+ AUTHENTICATION_SIMPLE);
Review comment:
Please keep indentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] frwrdt commented on a change in pull request
#3004: Support Amazon S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
frwrdt commented on a change in pull request #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#discussion_r384923635
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -511,7 +598,7 @@ public void pwrite(TBrokerFD fd, long offset, byte[] data) {
long currentStreamOffset;
try {
currentStreamOffset = fsDataOutputStream.getPos();
- } catch (IOException e) {
+ } catch (Exception e) {
Review comment:
should be 'IOException'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HangyuanLiu commented on a change in pull request
#3004: Support Amazon S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
HangyuanLiu commented on a change in pull request #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#discussion_r384554117
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -307,7 +343,58 @@ public BrokerFileSystem getFileSystem(String path, Map<String, String> propertie
fileSystem.getLock().unlock();
}
}
-
+
+ /**
+ * visible for test
+ *
+ * file system handle is cached, the identity is host + accessKey_secretKey
+ * @param path
+ * @param properties
+ * @return
+ * @throws URISyntaxException
+ * @throws Exception
+ */
+ public BrokerFileSystem getS3AFileSystem(String path, Map<String, String> properties) {
+ WildcardURI pathUri = new WildcardURI(path);
+ String accessKey = properties.containsKey(FS_S3A_ACCESS_KEY) ? properties.get(FS_S3A_ACCESS_KEY) : "";
Review comment:
properties.getOrDefault ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] frwrdt edited a comment on issue #3004: Support
Amazon S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
frwrdt edited a comment on issue #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#issuecomment-591799150
> Does access s3 need a SDK or dependent library?
> This library may be incorporated into third-party
There is no change in pom.xml and lib for be and fe.
This pr just add hadoop-aws sdk in pom.xml for broker and there is no conflict in dependencies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3004: Support Amazon
S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#issuecomment-591464914
Does access s3 need a SDK or dependent library?
This library may be incorporated into third-party
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] frwrdt commented on issue #3004: Support Amazon
S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
frwrdt commented on issue #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#issuecomment-591799150
> Does access s3 need a SDK or dependent library?
> This library may be incorporated into third-party
> Does access s3 need a SDK or dependent library?
> This library may be incorporated into third-party
There is no change in pom.xml and lib for be and fe.
This pr just add hadoop-aws sdk in pom.xml for broker and there is no conflict in dependencies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #3004: Support Amazon S3
data source in Broker Load
Posted by GitBox <gi...@apache.org>.
imay merged pull request #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] frwrdt commented on a change in pull request
#3004: Support Amazon S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
frwrdt commented on a change in pull request #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#discussion_r384923488
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -307,7 +343,58 @@ public BrokerFileSystem getFileSystem(String path, Map<String, String> propertie
fileSystem.getLock().unlock();
}
}
-
+
+ /**
+ * visible for test
+ *
+ * file system handle is cached, the identity is host + accessKey_secretKey
+ * @param path
+ * @param properties
+ * @return
+ * @throws URISyntaxException
+ * @throws Exception
+ */
+ public BrokerFileSystem getS3AFileSystem(String path, Map<String, String> properties) {
+ WildcardURI pathUri = new WildcardURI(path);
+ String accessKey = properties.containsKey(FS_S3A_ACCESS_KEY) ? properties.get(FS_S3A_ACCESS_KEY) : "";
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] frwrdt commented on a change in pull request
#3004: Support Amazon S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
frwrdt commented on a change in pull request #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#discussion_r384923421
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -146,12 +182,12 @@ public BrokerFileSystem getFileSystem(String path, Map<String, String> propertie
String password = properties.getOrDefault(PASSWORD_KEY, "");
String dfsNameServices = properties.getOrDefault(DFS_NAMESERVICES_KEY, "");
String authentication = properties.getOrDefault(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
- AUTHENTICATION_SIMPLE);
+ AUTHENTICATION_SIMPLE);
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HangyuanLiu commented on a change in pull request
#3004: Support Amazon S3 data source in Broker Load
Posted by GitBox <gi...@apache.org>.
HangyuanLiu commented on a change in pull request #3004: Support Amazon S3 data source in Broker Load
URL: https://github.com/apache/incubator-doris/pull/3004#discussion_r384552088
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -511,7 +598,7 @@ public void pwrite(TBrokerFD fd, long offset, byte[] data) {
long currentStreamOffset;
try {
currentStreamOffset = fsDataOutputStream.getPos();
- } catch (IOException e) {
+ } catch (Exception e) {
Review comment:
Why catch all exception
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org