[jira] [Created] (HUDI-4728) Add support to skip larger log blocks with minor log compaction

["sivabalan narayanan (Jira)" <ji...@apache.org>]

sivabalan narayanan created HUDI-4728:
-----------------------------------------

             Summary: Add support to skip larger log blocks with minor log compaction
                 Key: HUDI-4728
                 URL: https://issues.apache.org/jira/browse/HUDI-4728
             Project: Apache Hudi
          Issue Type: Improvement
          Components: compaction
            Reporter: sivabalan narayanan


Is there a size threshold to exclude big log blocks? Why do log compaction on log blocks that are big enough? Thoughts

 

Good point. For initial version we want to target all the blocks. In the coming iterations I will include block sizes threshold as well. Current logic of AbstractHoodieLogRecordReader should be able to handle it as well.

 

For streaming workloads, this might be very heavy. So, we need to support this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)