[GitHub] [incubator-doris] qidaye opened a new issue #8807: [Feature] Support S3 reource

[GitBox <gi...@apache.org>]

qidaye opened a new issue #8807:
URL: https://github.com/apache/incubator-doris/issues/8807


   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Description
   
   ## Background
   
   To support the separation of hot and cold data, you need to specify the connection information of remote storage S3, such as ak, sk, url, etc., when creating an OLAP table. 
   This information can be reused, and a Doris cluster may also be connected to multiple S3s, so these connection information needs to be managed as a remote storage resource. It is convenient for users to use, and does not need to be provided every time a table is created, only the resource name is required. 
   At the same time, when the remote storage connection information changes, there is no need to rebuild the table, just update the corresponding resource.
   
   ## Design
   
   Currently, Doris supports the creation of Spark and MySQL resources, and the corresponding syntax can be reused. Based on the existing Resource abstraction, Resource manager resource meta-information management class,
   
   1. Add s3 type to ResourceType
   2. Create a new S3Resource class, inherit the Resource class, and implement the corresponding interface
   3. Support the verification, reading and persistence of s3 resource information
   
   ### Specific syntax
   
   ```sql
   -- create remote storage resource
   create resource "remote_s3"
   properties
   (
      "type" = "s3",
      "s3_endpoint" = "bj",
      "s3_region" = "bj",
      "s3_root_path" = "/path/to/root",
      "s3_access_key" = "bbb",
      "s3_secret_key" = "aaaa",
      "s3_max_connections" = "50",
      "s3_request_timeout_ms" = "3000",
      "s3_connection_timeout_ms" = "1000"
   );
   
   
   -- create table using remote storage resource
   CREATE TABLE example_db.table_remote
       (
       k1 DATE,
       k2 INT,
       k3 SMALLINT,
       k4 VARCHAR(2048),
       k5 DATETIME
       )
       ENGINE=olap
       PROPERTIES
       (
       "storage_medium" = "HDD",
       "storage_cold_medium" = "S3",
       "remote_storage_resource" = "remote_s3",
       "storage_cooldown_time" = "2022-06-04 00:00:00"
       );
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org