[GitHub] [pinot] xiangfu0 commented on pull request #8465: Add adhoc minion task creation endpoint

[GitBox <gi...@apache.org>]

xiangfu0 commented on PR #8465:
URL: https://github.com/apache/pinot/pull/8465#issuecomment-1089067078

   > Please give me a couple of days, I would like to review this.
   > 
   > Meanwhile, if you can please mention some use cases I would appreciate it.
   
   Thanks, @mcvsubbu, In short, the goal here is to add a new public endpoint to schedule ad-hoc minion tasks.
   
   Current minion tasks require all tasks configs to be part of table configs, it makes sense for regularly scheduled jobs, but when I want to quickly test something, it's hard for me to bring in data using minion.
   
   Once this feature is in place, we can also implement the feature of SQL `INSERT INTO` with raw files in the query console to simplify the ingestion process. 
   
   E.g.
   ```
   INSERT INTO [db.]table [(c1, c2, c3)] FROM INFILE file_name [COMPRESSION type] FORMAT format_name;
   ```
   or 
   ```
   LOAD LABEL [db.]task (
        DATA INFILE("s3a://my-s3-bucket/*/*.parquet") INTO TABLE `table`
   )
   WITH MINION minionTag (
     "fs.s3.accessKeyId" = "<my-access-key>",
     "fs.s3.accessKeySecret" = "<my-secret-key>"
   );
   ```
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org