You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/01/13 11:35:02 UTC
[GitHub] [incubator-doris] yangzhg opened a new issue #5232: [Proposal] Supports direct reading and writing of hdfs and cloud storage without a broker
yangzhg opened a new issue #5232:
URL: https://github.com/apache/incubator-doris/issues/5232
Using broker to read hdfs and bos Although the implementation of different file systems is shielded, there are the following problems:
1. The process of reading and writing interacts with the broker too much through the thrift interface, which increases the probability of errors
2. The performance of reading via broker is worse than direct reading due to transit
3. The method of uploading files through the broker is to first upload to the temporary file and then rename it into the final file. In many cloud storages, renaming is not supported. Some of the methods of copy+delete are real copying rather than simple modification. Data is a waste of time, especially when there are more files to upload
Therefore, we intend to support direct access through hdfs and s3 protocols, and support most cloud storage through s3 protocols, such as baidu bos, aliyun oss, etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg closed issue #5232: [Proposal] Supports direct reading and writing of hdfs and cloud storage without a broker
Posted by GitBox <gi...@apache.org>.
yangzhg closed issue #5232:
URL: https://github.com/apache/incubator-doris/issues/5232
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org