[GitHub] [incubator-doris] imay commented on issue #3549: [Proposal] Support `SELECT INTO OUTFILE` feature.

[GitBox <gi...@apache.org>]

imay commented on issue #3549:
URL: https://github.com/apache/incubator-doris/issues/3549#issuecomment-626520097


   @morningman 
   Wonderful proposal.
   I have some questions about this.
   
   > Write the result to the local file system of a BE. And provide users with an http connection for downloading.
   
   If the result file will be downloaded through HTTP, what's the authentication information will be attached for the download request. If there is an authentication check for the download request, the relation between user and result file will be saved. Maybe through rule, such as all result belong to one user locates in the same directory. Or FE will store all the authentication information.
   
   Besides that, it is not a clever idea to store huge data in one single local file. So we should have a solution to support split a huge result into many files in local filesystem. And if it is supported, what user gets is not a single URL, he will get a list of URLs.
   
   And there is another choice that we don't support unload to local file, we only can unload file to remote storage, such as S3, HDFS.
   
   > {"download_url" : "http://be:http_port/file.."}
   
   1. I think download operation should be routed by FE. And we can't give user the absolute path in the URL. It is a good that FE can rewrite the path.
   2. Usually unload is a heavy operation, if user lose the returned download URL, what left for him/her is to do this unload again, which is not efficient. If we make sure to support unload to local file, what I suggest is that we should support a command for users to list all unload result information.
   3. As far as I know, it is hard to get information from OK message of MySQL protocol. It is not a good design to return load-id for `insert into select` command. So I think we should avoid this in unload operation.
   
   > `result_size_in_bytes`
   I think "max_result_bytes" is better.
   
   > `result_file_expiration_time`
   `result_expiration_second`. Better to give unit information in name.
   
   > `file name`
   fix typo to `name_prefix` and I think there is no need to add TTL in file name.
   
   After all, I think it is a good way to support remote FS first, because it will avoid us to do many things unrelated.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org