[GitHub] [hudi] nsivabalan commented on issue #6943: [SUPPORT] distinct at ColumnStatIndexSupport.scala:141

[GitBox <gi...@apache.org>]

nsivabalan commented on issue #6943:
URL: https://github.com/apache/hudi/issues/6943#issuecomment-1283146480

   I remember you telling me that you had 500+ columns. Its known that you could see a perf hit w/ such large columns and since hudi builds col stats index for all columns. 
   
   Suggestion here is to disable col stats and re-build w/ just a few cols.
   Step1: 
   you can disable col stats fully hoodie.metadata.index.column.stats.enable = false. So during your next write, hudi will clean up entire col stats partition in metadata table.
   Step2: after few commits, you can re-enable col stats for your table, by explicitly setting cols to index.
   `hoodie.metadata.index.column.stats.enable=true`
   and by setting appropriate value for
   hoodie.metadata.index.column.stats.column.list
   your first commit with this new set of configs might hav e some higher latency since it bootstraps entire col stats for your table.
   
   After this, you should be able to leverage col stats in your queries. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org