You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/03/30 17:03:32 UTC

[GitHub] [incubator-pinot] fx19880617 opened a new issue #6729: Memory Size Estimation on UpsertTable

fx19880617 opened a new issue #6729:
URL: https://github.com/apache/incubator-pinot/issues/6729


   We need tooling to estimate the memory/disk size for the upsert table based on primary keys and their cardinalities.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] fx19880617 commented on issue #6729: Memory Size Estimation on UpsertTable

Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6729:
URL: https://github.com/apache/incubator-pinot/issues/6729#issuecomment-810468145


   This is to have the tooling to estimate upfront size, just estimate.
   E.g. ```pinot-admin.sh EstimateTableSize -schema mySchema.json -tableConf myTable.json -columnStats xxxx```
   @deemoliu
   
   E.g. I have a composite primary key with two string columns, estimated size is 24 bytes and 40 bytes, the unique combination is 16 million. 
   Then for the Primary keymap size estimation:
   - For keys:  keySpace is (24+40)Bytes* 16 Million = 1GB
   - For values: we need to store RecordLocation, it contains <segmentName, DocId, Timestamp>, so maybe another <48+ 4 + 8>=64 bytes on it. It's another 1GB.
   
   Then we can estimate that the total memory overhead is about 2GB.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] yupeng9 commented on issue #6729: Memory Size Estimation on UpsertTable

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on issue #6729:
URL: https://github.com/apache/incubator-pinot/issues/6729#issuecomment-810456227


   https://github.com/apache/incubator-pinot/pull/6604 is supposed to address this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] fx19880617 edited a comment on issue #6729: Memory Size Estimation on UpsertTable

Posted by GitBox <gi...@apache.org>.
fx19880617 edited a comment on issue #6729:
URL: https://github.com/apache/incubator-pinot/issues/6729#issuecomment-810468145


   This is to have the tooling to estimate upfront size, just estimate.
   E.g. ```pinot-admin.sh EstimateTableSize -schema mySchema.json -tableConf myTable.json -columnStats xxxx```
   @deemoliu
   
   E.g. I have a composite primary key with two string columns, estimated size is 24 bytes and 40 bytes, the unique combination is 16 million. 
   Then for the Primary keymap size estimation(Per https://sourcegraph.com/github.com/apache/incubator-pinot@89a22f097c5ff26396e58950c90d764066a56121/-/blob/pinot-core/src/main/java/org/apache/pinot/core/upsert/PartitionUpsertMetadataManager.java#L72):
   - For keys:  keySpace is (24+40)Bytes* 16 Million = 1GB
   - For values: we need to store RecordLocation, it contains <segmentName, DocId, Timestamp>, so maybe another <48+ 4 + 8>=64 bytes on it. It's another 1GB.
   
   Then we can estimate that the total memory overhead is about 2GB.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] yupeng9 commented on issue #6729: Memory Size Estimation on UpsertTable

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on issue #6729:
URL: https://github.com/apache/incubator-pinot/issues/6729#issuecomment-810535987


   I see. This will be a useful tool to add. Also, we can leverage the metrics for calibration/validation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org