[GitHub] [pinot] mcvsubbu commented on issue #7192: Flush Segments on a Schedule

[GitBox <gi...@apache.org>]

mcvsubbu commented on issue #7192:
URL: https://github.com/apache/pinot/issues/7192#issuecomment-885897977


   Repeating my message to @mapshen  in the slack channel:
   Please re-run the realtime prov helper  and paste the output in this issue.
   
   There is no standard for the time taken to build a segment. It is a function of   how large your segment is (which includes  number of rows, cardinality of columns, number of columns, length of string columns, the type and number of indices you have in there, etc.). In general, DO NOT use number of rows as the limit. It is better to use segment size as the limit. I hope you are doing that. If so, can you share what the size is, and can you reduce the size? It is also possible that all partitions on the machine are completing the segment at the same time. How many partitions do you have on any one machine? It is possible that your machine is (a) swapping or (b) GC-ing or both. What is your heap size?  Also, what is your QPS? How much delay can you tolerate for incoming freshness of data? Do you know that Pinot provides a way to measure the freshness of data in your query response? 
   
   There are knobs that you can tune before starting to propose protocol changes (that can cause unnecessary race conditions in the code and issues for other installations). 
   
   What is your ingestion rate (per partition)? both in number of messages and number of bytes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org