[GitHub] [hudi] nsivabalan commented on issue #7657: [SUPPORT] Invalid number of file groups for partition:column_stats

[GitBox <gi...@apache.org>]

nsivabalan commented on issue #7657:
URL: https://github.com/apache/hudi/issues/7657#issuecomment-1396365792

   there are some issues wrt some of the metadata configs. Some of the metadata table configs are not meant to be overridden. 
   
   --hoodie-conf hoodie.metadata.compact.max.delta.commits=10 \
   --hoodie-conf hoodie.metadata.cleaner.commits.retained=60 \
   --hoodie-conf hoodie.metadata.populate.meta.fields=true \
   --hoodie-conf hoodie.metadata.clean.async=true \
   --hoodie-conf hoodie.metadata.keep.max.commits=80 \
   --hoodie-conf hoodie.metadata.insert.parallelism=20 \
   --hoodie-conf hoodie.metadata.keep.min.commits=70 \
   
   Please do not override any of these configs. I have created a follow up ticket to not expose these configs
   https://issues.apache.org/jira/browse/HUDI-5582
   
   Can you restart metadata from scratch w/ right set of configs and let us know how it goes. 
   But I have some clarification as well. I see you are setting async indexing for metadata table to true. So, are you spinning up a separate spark submit job (other than deltastreamer) to build the metadata indexes. If not, you may have to fix that as well. More details can be found here https://hudi.apache.org/docs/metadata_indexing
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org