[GitHub] [pinot] bdstuart commented on issue #8014: RealtimeToOfflineSegmentsTask tasks time out with large amount of rows

[GitBox <gi...@apache.org>]

bdstuart commented on issue #8014:
URL: https://github.com/apache/pinot/issues/8014#issuecomment-1204380117

   @kishoreg this is a similar issues to what we are experiencing during our RealtimeToOfflineSegmentsTask. It's taking almost 8 hours to process 175m rows of data. 
   Initialized mapper with 56 record readers (takes almost 2 hours)
   Start sorting on numRows: 172432000, numSortFields: 44 takes Finish sorting in 10563254ms (takes almost 3 hours)
   Finish creating rollup file in 5990480ms (takes about 1.5 hours)
   
   We are using one minion that is 4cpu/32gb ram/300gb storage and it's running on one thread it seems.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org