[GitHub] [incubator-kyuubi] ulysses-you commented on issue #1299: [FEATURE] Support merge small files sql

[GitBox <gi...@apache.org>]

ulysses-you commented on issue #1299:
URL: https://github.com/apache/incubator-kyuubi/issues/1299#issuecomment-952682819


   thank you @melin .
   
   Kyuubi has already supported merge small file during writing by adding an extra shuffle, can see [here](https://github.com/apache/incubator-kyuubi/blob/master/dev/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/RepartitionBeforeWritingBase.scala). So I believe it's easy to do the similar things using Kyuubi by:
   ```sql
   INSERT OVERWRITE TABLE $table AS SELECT * FROM $table;
   ```
   
   For this case, we probably can enhance the existed rule to support `fileCount` and `maxRecordsPerFile` (which is this issue want to do) by using `parquet` or `orc` footer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org