You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/02/23 23:26:58 UTC
[GitHub] [incubator-pinot] deemoliu opened a new pull request #6604: emit metric for upsert table memory usage
deemoliu opened a new pull request #6604:
URL: https://github.com/apache/incubator-pinot/pull/6604
## Description
#6576 Emit metric for upsert table metadata (store mapping from primaryKey to RecordLocation) memory usage in bytes.
[Previous work](https://github.com/apache/incubator-pinot/pull/6204/files#diff-2fc79d140ad00f4fe862b03a12674c564a3cd85bc59e5397cb706a51a19faa39R46 ) emitting metric for PK numbers.
## Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
No.
Does this PR fix a zero-downtime upgrade introduced earlier?
No.
Does this PR otherwise need attention when creating release notes? Things to consider:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
No.
## Release Notes
If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release.
this is a backward compatible change.
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
## Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #6604: emit metric for upsert table memory usage
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on a change in pull request #6604:
URL: https://github.com/apache/incubator-pinot/pull/6604#discussion_r581476233
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/upsert/PartitionUpsertMetadataManager.java
##########
@@ -22,6 +22,7 @@
import java.util.Iterator;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.concurrent.ThreadSafe;
+import jdk.nashorn.internal.ir.debug.ObjectSizeCalculator;
Review comment:
Is this supported in all JDK versions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6604: emit metric for upsert table memory usage
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6604:
URL: https://github.com/apache/incubator-pinot/pull/6604#discussion_r583100402
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/upsert/PartitionUpsertMetadataManager.java
##########
@@ -22,6 +22,7 @@
import java.util.Iterator;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.concurrent.ThreadSafe;
+import jdk.nashorn.internal.ir.debug.ObjectSizeCalculator;
Review comment:
Also, please make sure the map size calculation is very fast because it happens on a per-record basis
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #6604: emit metric for upsert table memory usage
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on a change in pull request #6604:
URL: https://github.com/apache/incubator-pinot/pull/6604#discussion_r581475438
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java
##########
@@ -43,6 +43,7 @@
LLC_SIMULTANEOUS_SEGMENT_BUILDS("llcSimultaneousSegmentBuilds", true),
// Upsert metrics
+ UPSERT_METADATA_MEMORY_USED("bytes", false),
Review comment:
UPSERT_METADATA_MEMORY_USED("upsertMetadataMemoryUsed", false),
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #6604: emit metric for upsert table memory usage
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on a change in pull request #6604:
URL: https://github.com/apache/incubator-pinot/pull/6604#discussion_r582265022
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/upsert/PartitionUpsertMetadataManager.java
##########
@@ -22,6 +22,7 @@
import java.util.Iterator;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.concurrent.ThreadSafe;
+import jdk.nashorn.internal.ir.debug.ObjectSizeCalculator;
Review comment:
I somehow feel you may need to add/minus the bytes when there is a primary add/delete/update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] deemoliu commented on a change in pull request #6604: emit metric for upsert table memory usage
Posted by GitBox <gi...@apache.org>.
deemoliu commented on a change in pull request #6604:
URL: https://github.com/apache/incubator-pinot/pull/6604#discussion_r582239046
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java
##########
@@ -43,6 +43,7 @@
LLC_SIMULTANEOUS_SEGMENT_BUILDS("llcSimultaneousSegmentBuilds", true),
// Upsert metrics
+ UPSERT_METADATA_MEMORY_USED("bytes", false),
Review comment:
Gotcha, thanks for pointing out!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] deemoliu commented on a change in pull request #6604: emit metric for upsert table memory usage
Posted by GitBox <gi...@apache.org>.
deemoliu commented on a change in pull request #6604:
URL: https://github.com/apache/incubator-pinot/pull/6604#discussion_r582238968
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/upsert/PartitionUpsertMetadataManager.java
##########
@@ -22,6 +22,7 @@
import java.util.Iterator;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.concurrent.ThreadSafe;
+import jdk.nashorn.internal.ir.debug.ObjectSizeCalculator;
Review comment:
Hi @fx19880617 I found ObjectSizeCalculator is not supported JDK11.
https://mail.openjdk.java.net/pipermail/jdk8u-dev/2020-May/011770.html
I also looked into calculating the bytes used by metadata manually, but it's not optimal since it required extra storage.
Do you have any suggestions or recommendations, is there any similar metrics emitted by Pinot?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org