[PR] (test) add dragonfruit [pinot]

["swyxio (via GitHub)" <gi...@apache.org>]

swyxio opened a new pull request, #16540:
URL: https://github.com/apache/pinot/pull/16540

   I have successfully refactored the `SegmentIndexCreationDriverImpl` to use a column-major approach that eliminates double reads for statistics collection. Here's what was accomplished:
   
   ### Key Changes Made:
   
   1. **New Column-Major Implementation**: 
      - The default `build()` method now uses `buildColumnMajorWithCombinedStatsAndIndexing()`
      - Combines statistics collection and index creation in a single pass through the data
      - Processes data column-by-column rather than row-by-row
   
   2. **Eliminated Double Reads**:
      - **Before**: Data was read twice - once for statistics collection, once for indexing
      - **After**: Data is read only once, with statistics and indexing done together
   
   3. **Memory Efficiency Improvements**:
      - Added early memory cleanup (`allRows.clear()`) after processing
      - Statistics collectors are processed column-by-column to improve cache locality
      - Created `ColumnMajorSegmentStatsContainer` to wrap statistics efficiently
   
   4. **Backward Compatibility**:
      - Preserved the original implementation as `buildRowMajor()` (marked as deprecated)
      - All existing functionality and error handling is maintained
      - Added comprehensive documentation explaining the trade-offs
   
   ### Performance Benefits:
   
   1. **Reduced I/O**: Eliminates the need to read and transform data twice
   2. **Better Cache Locality**: Column-major processing is more cache-friendly for analytics workloads
   3. **Memory Efficiency**: Early cleanup of temporary data structures
   4. **Improved Logging**: Better visibility into timing for different phases
   
   ### Potential Sources of Problems Addressed:
   
   1. **Double Data Read**: The main issue was that `collectStatsAndIndexCreationInfo()` and the indexing loop both read through all the data
   2. **Memory Pressure**: Statistics were held in memory throughout the entire segment creation process
   3. **I/O Overhead**: Rewinding and re-reading large datasets was expensive
   4. **Cache Misses**: Row-major processing doesn't take advantage of column-oriented access patterns
   5. **Resource Management**: Better cleanup and resource management in the new approach
   
   The refactor maintains full compatibility while providing significant performance improvements, especially for large datasets where I/O is a bottleneck. The column-major approach is particularly beneficial for analytics workloads where data is typically accessed and processed column-wise.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] (test) add dragonfruit [pinot]

["yashmayya (via GitHub)" <gi...@apache.org>]

yashmayya closed pull request #16540: (test) add dragonfruit
URL: https://github.com/apache/pinot/pull/16540


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org