[GitHub] [incubator-pinot] fx19880617 edited a comment on issue #6729: Memory Size Estimation on UpsertTable

[GitBox <gi...@apache.org>]

fx19880617 edited a comment on issue #6729:
URL: https://github.com/apache/incubator-pinot/issues/6729#issuecomment-810468145


   This is to have the tooling to estimate upfront size, just estimate.
   E.g. ```pinot-admin.sh EstimateTableSize -schema mySchema.json -tableConf myTable.json -columnStats xxxx```
   @deemoliu
   
   E.g. I have a composite primary key with two string columns, estimated size is 24 bytes and 40 bytes, the unique combination is 16 million. 
   Then for the Primary keymap size estimation(Per https://sourcegraph.com/github.com/apache/incubator-pinot@89a22f097c5ff26396e58950c90d764066a56121/-/blob/pinot-core/src/main/java/org/apache/pinot/core/upsert/PartitionUpsertMetadataManager.java#L72):
   - For keys:  keySpace is (24+40)Bytes* 16 Million = 1GB
   - For values: we need to store RecordLocation, it contains <segmentName, DocId, Timestamp>, so maybe another <48+ 4 + 8>=64 bytes on it. It's another 1GB.
   
   Then we can estimate that the total memory overhead is about 2GB.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org