You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/03/12 23:27:34 UTC
[GitHub] [incubator-pinot] mqliang opened a new pull request #6680: Instrument combine operators with total thread time
mqliang opened a new pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680
## Description
Instrumentation is conducted at BaseCombineOperator, so all combine operator extends BaseCombineOperator will get the instrumentation(unless it override the getNextBlock() function)
## Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
* [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR fix a zero-downtime upgrade introduced earlier?
* [ ] Yes (Please label this as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR otherwise need attention when creating release notes? Things to consider:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
* [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes)
## Release Notes
If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release.
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
## Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r594783507
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java
##########
@@ -44,6 +44,7 @@
String REQUEST_ID_METADATA_KEY = "requestId";
String NUM_RESIZES_METADATA_KEY = "numResizes";
String RESIZE_TIME_MS_METADATA_KEY = "resizeTimeMs";
+ String THREAD_TIME_MS = "threadTimeMs";
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-797826488
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=h1) Report
> Merging [#6680](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=desc) (2b8178b) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `0.57%`.
> The diff coverage is `62.64%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6680 +/- ##
==========================================
- Coverage 66.44% 65.87% -0.58%
==========================================
Files 1075 1389 +314
Lines 54773 67539 +12766
Branches 8168 9791 +1623
==========================================
+ Hits 36396 44489 +8093
- Misses 15700 19890 +4190
- Partials 2677 3160 +483
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.87% <62.64%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...pinot/broker/api/resources/PinotClientRequest.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdENsaWVudFJlcXVlc3QuamF2YQ==) | `0.00% <0.00%> (-27.28%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <ø> (+9.52%)` | :arrow_up: |
| [...n/java/org/apache/pinot/client/BrokerResponse.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Jyb2tlclJlc3BvbnNlLmphdmE=) | `100.00% <ø> (ø)` | |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <ø> (-13.29%)` | :arrow_down: |
| [...org/apache/pinot/client/DynamicBrokerSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0R5bmFtaWNCcm9rZXJTZWxlY3Rvci5qYXZh) | `82.85% <ø> (+10.12%)` | :arrow_up: |
| [...n/java/org/apache/pinot/client/ExecutionStats.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0V4ZWN1dGlvblN0YXRzLmphdmE=) | `68.88% <ø> (ø)` | |
| ... and [1275 more](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=footer). Last update [656f279...371d3ec](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-797826488
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=h1) Report
> Merging [#6680](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=desc) (86e8cdd) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `22.89%`.
> The diff coverage is `42.85%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6680 +/- ##
===========================================
- Coverage 66.44% 43.55% -22.90%
===========================================
Files 1075 1391 +316
Lines 54773 67585 +12812
Branches 8168 9794 +1626
===========================================
- Hits 36396 29436 -6960
- Misses 15700 35681 +19981
+ Partials 2677 2468 -209
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `43.55% <42.85%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...t/broker/broker/BasicAuthAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0Jhc2ljQXV0aEFjY2Vzc0NvbnRyb2xGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/pinot/broker/queryquota/QueryQuotaEntity.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9RdWVyeVF1b3RhRW50aXR5LmphdmE=) | `0.00% <0.00%> (-50.00%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ceselector/StrictReplicaGroupInstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL1N0cmljdFJlcGxpY2FHcm91cEluc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...roker/routing/segmentpruner/TimeSegmentPruner.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1RpbWVTZWdtZW50UHJ1bmVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...roker/routing/segmentpruner/interval/Interval.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL2ludGVydmFsL0ludGVydmFsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| ... and [1418 more](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=footer). Last update [d1cf36a...86e8cdd](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595585739
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
##########
@@ -317,6 +317,9 @@
public static final String ACCESS_CONTROL_FACTORY_CLASS = "pinot.server.admin.access.control.factory.class";
public static final String DEFAULT_ACCESS_CONTROL_FACTORY_CLASS =
"org.apache.pinot.server.api.access.AllowAllAccessFactory";
+
+ public static final String CONFIG_OF_ENABLE_THREAD_CPU_TIME_INSTRUMENT = "pinot.server.instance.enableThreadCpuTimeInstrument";
Review comment:
(nit) rename this to pinot.server.instance.enableThreadCpuTimeMeasurement
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-797811858
@siddharthteotia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia merged pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
siddharthteotia merged pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596287394
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java
##########
@@ -65,6 +65,7 @@
SEGMENT_DOWNLOAD_FAILURES("segments", false),
NUM_RESIZES("numResizes", false),
RESIZE_TIME_MS("resizeTimeMs", false),
+ THREAD_CPU_TIME_NS("threadCpuTimeNs", false),
Review comment:
Should we also make NUM_RESIZES as ServerGauge? Or just RESIZE_TIME_MS?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-798779528
Let us settle to use nanosecond resolution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang edited a comment on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang edited a comment on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-800831123
@siddharthteotia rebased and all comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596567877
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/operator/combine/CombineSlowOperatorsTest.java
##########
@@ -65,7 +65,8 @@ public void setUp() {
public void testSelectionOnlyCombineOperator() {
List<Operator> operators = getOperators();
SelectionOnlyCombineOperator combineOperator = new SelectionOnlyCombineOperator(operators,
- QueryContextConverterUtils.getQueryContextFromPQL("SELECT * FROM table"), _executorService, TIMEOUT_MS);
+ QueryContextConverterUtils.getQueryContextFromPQL("SELECT * FROM table"),
Review comment:
I meant this change can be undone right?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java
##########
@@ -77,6 +77,7 @@
private PlanMaker _planMaker;
private long _defaultTimeOutMs = CommonConstants.Server.DEFAULT_QUERY_EXECUTOR_TIMEOUT_MS;
private ServerMetrics _serverMetrics;
+ boolean _enableThreadCpuTimeInstrument;
Review comment:
This can be removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595615949
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -89,12 +92,28 @@ protected IntermediateResultsBlock getNextBlock() {
_futures[i] = _executorService.submit(new TraceRunnable() {
@Override
public void runJob() {
+ ThreadTimer processThreadTimer = new ThreadTimer();
+ processThreadTimer.start();
+
processSegments(threadIndex);
+
+ processThreadTimer.stop();
+ totalWorkerTime.addAndGet(processThreadTimer.getThreadTime());
}
});
}
+ ThreadTimer mergeThreadTimer = new ThreadTimer();
Review comment:
cc @mcvsubbu @mqliang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-797884568
> > We should emit the metric like other server metrics and also see if the measurement can be made configurable via instance level config.
>
> This is a query level metric. If we emit it as BrokerMetrics/BrokerGauge, the number of metrics will explode. But we can expose it as a part of query response metadata, just like numServersQueried/numDocsScanned).
I don't follow. I am not sure how BrokerMetric or BrokerGauge is involved here in the server context. From the server we should emit this query level metric and we should also include this as part of data table metadata (numDocsScanned etc) that is sent to the broker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r593513263
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -89,12 +92,28 @@ protected IntermediateResultsBlock getNextBlock() {
_futures[i] = _executorService.submit(new TraceRunnable() {
@Override
public void runJob() {
+ ThreadTimer processThreadTimer = new ThreadTimer();
+ processThreadTimer.start();
+
processSegments(threadIndex);
+
+ processThreadTimer.stop();
+ totalWorkerTime.addAndGet(processThreadTimer.getThreadTime());
}
});
}
+ ThreadTimer mergeThreadTimer = new ThreadTimer();
Review comment:
I don't think we should instrument the merge part this way.
We should do that in InstanceResponseOperator and InstanceResponseBlock because that is the single main thread and all the merge work is happening under that thread.
So we start the main thread timer in InstanceResponseOperator and stop the main thread timer in InstanceResponseBlock's constructor and then add up the timer received from merged intermediate result block. This will cover both single threaded and multi-threaded code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595642761
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/InstanceResponseBlock.java
##########
@@ -41,6 +41,9 @@
public InstanceResponseBlock(IntermediateResultsBlock intermediateResultsBlock) {
try {
_instanceResponseDataTable = intermediateResultsBlock.getDataTable();
+ long totalThreadCpuTimeNs = intermediateResultsBlock.getThreadCpuTimeNs();
+ _instanceResponseDataTable.getMetadata().put(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY,
Review comment:
If we don't set the value as DataTable metadata here, then it's impossible to get the value in QueryScheduler.java, in other words, we should delete the ServerGauge emitting code also. I think at this moment, we can just put a log line here, and deleting the change in QueryScheduler.java.
Another workaround is: add the value as DataTable metadata here, but in QueryScheduler.java, after writing log and emitting ServerGauge, `dataTable.getMetadata().remove(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY)` so that this value will NOT be send back to broker (and add a TODO stating remove it when we have data table serialization cost measuring).
Any preference between the two option?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596289528
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java
##########
@@ -65,6 +65,7 @@
SEGMENT_DOWNLOAD_FAILURES("segments", false),
NUM_RESIZES("numResizes", false),
RESIZE_TIME_MS("resizeTimeMs", false),
+ THREAD_CPU_TIME_NS("threadCpuTimeNs", false),
Review comment:
Only RESIZE_TIME
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595638644
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/InstanceResponseBlock.java
##########
@@ -41,6 +41,9 @@
public InstanceResponseBlock(IntermediateResultsBlock intermediateResultsBlock) {
try {
_instanceResponseDataTable = intermediateResultsBlock.getDataTable();
+ long totalThreadCpuTimeNs = intermediateResultsBlock.getThreadCpuTimeNs();
+ _instanceResponseDataTable.getMetadata().put(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY,
Review comment:
I agree. Let's send the value from server to broker in the DataTable metadata (or whatever) only after we do the follow-up of measuring the data table serialization cost. cc @mqliang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r593793706
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java
##########
@@ -44,6 +44,7 @@
String REQUEST_ID_METADATA_KEY = "requestId";
String NUM_RESIZES_METADATA_KEY = "numResizes";
String RESIZE_TIME_MS_METADATA_KEY = "resizeTimeMs";
+ String THREAD_TIME_MS = "threadTimeMs";
Review comment:
Why should this be changed to microseconds?
The jmx api documentation says it returns nanosecond precision (but may not be necessarily accurate). But we can't treat it as microseconds.
For example, if the returned value is
1000000. It is 1000 microseconds. If we treat the returned value as microseconds, then we are increasing the value by 1000 times making it 1000ms.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596482255
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/operator/combine/CombineSlowOperatorsTest.java
##########
@@ -65,7 +65,8 @@ public void setUp() {
public void testSelectionOnlyCombineOperator() {
List<Operator> operators = getOperators();
SelectionOnlyCombineOperator combineOperator = new SelectionOnlyCombineOperator(operators,
- QueryContextConverterUtils.getQueryContextFromPQL("SELECT * FROM table"), _executorService, TIMEOUT_MS);
+ QueryContextConverterUtils.getQueryContextFromPQL("SELECT * FROM table"),
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r594783750
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -89,12 +92,28 @@ protected IntermediateResultsBlock getNextBlock() {
_futures[i] = _executorService.submit(new TraceRunnable() {
@Override
public void runJob() {
+ ThreadTimer processThreadTimer = new ThreadTimer();
Review comment:
done
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/IntermediateResultsBlock.java
##########
@@ -69,6 +69,7 @@
private boolean _numGroupsLimitReached;
private int _numResizes;
private long _resizeTimeMs;
+ private long _threadTime;
Review comment:
done
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -89,12 +92,28 @@ protected IntermediateResultsBlock getNextBlock() {
_futures[i] = _executorService.submit(new TraceRunnable() {
@Override
public void runJob() {
+ ThreadTimer processThreadTimer = new ThreadTimer();
+ processThreadTimer.start();
+
processSegments(threadIndex);
+
+ processThreadTimer.stop();
+ totalWorkerTime.addAndGet(processThreadTimer.getThreadTime());
Review comment:
done
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isThreadCpuTimeSupported();
+ private static final long NANOS_IN_MILLIS = 1;
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+
+ private long _startTimeMs = -1;
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596482320
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,76 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code ThreadTimer} class providing the functionality of measuring the CPU time for the current thread.
+ */
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_CURRENT_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isCurrentThreadCpuTimeSupported();
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+ private static boolean IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED;
+ private long _startTimeNs = -1;
+ private long _endTimeNs = -1;
+
+ public ThreadTimer() {
+ }
+
+ public static boolean isCurrentThreadCpuTimeSupported() {
+ return IS_CURRENT_THREAD_CPU_TIME_SUPPORTED;
+ }
+
+ public void start() {
+ if (IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED && IS_CURRENT_THREAD_CPU_TIME_SUPPORTED) {
+ _startTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+ }
+ }
+
+ public void stop() {
+ if (IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED && IS_CURRENT_THREAD_CPU_TIME_SUPPORTED) {
+ _endTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+ }
+ }
+
+ public long getThreadTimeNs() {
+ if (_startTimeNs == -1 || _endTimeNs == -1) {
+ return 0;
+ }
+ return _endTimeNs - _startTimeNs;
+ }
+
+ public long stopAndGetThreadTimeNs() {
+ stop();
+ return getThreadTimeNs();
+ }
+
+ static {
+ LOGGER.info("Current thread cpu time measurement supported: {}", IS_CURRENT_THREAD_CPU_TIME_SUPPORTED);
+ }
+
+ public static void setThreadCpuTimeMeasurementEnabled(boolean enable) {
+ IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED = enable;
Review comment:
done
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/InstanceResponseBlock.java
##########
@@ -41,6 +41,9 @@
public InstanceResponseBlock(IntermediateResultsBlock intermediateResultsBlock) {
try {
_instanceResponseDataTable = intermediateResultsBlock.getDataTable();
+ long totalThreadCpuTimeNs = intermediateResultsBlock.getThreadCpuTimeNs();
+ _instanceResponseDataTable.getMetadata().put(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY,
Review comment:
done
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
##########
@@ -317,6 +317,9 @@
public static final String ACCESS_CONTROL_FACTORY_CLASS = "pinot.server.admin.access.control.factory.class";
public static final String DEFAULT_ACCESS_CONTROL_FACTORY_CLASS =
"org.apache.pinot.server.api.access.AllowAllAccessFactory";
+
+ public static final String CONFIG_OF_ENABLE_THREAD_CPU_TIME_INSTRUMENT = "pinot.server.instance.enableThreadCpuTimeInstrument";
Review comment:
done
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java
##########
@@ -65,6 +65,7 @@
SEGMENT_DOWNLOAD_FAILURES("segments", false),
NUM_RESIZES("numResizes", false),
RESIZE_TIME_MS("resizeTimeMs", false),
+ THREAD_CPU_TIME_NS("threadCpuTimeNs", false),
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-798982231
> Just to clarify:
> The value returned is in nanos. Initially in the prototype I had this converted to milli seconds, and most values turned out to be 0 when dividing nanos by 1_000_000 to get millis. So I changed it to log the values in nanos, and found that the resolution was much better (i;.e. we get non-zero numbers). It is ok to convert to microseconds. Or just leave it at nanoseconds. But converting to millis will not work
Agreed. I also prefer nano resolution here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r597093906
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code ThreadTimer} class providing the functionality of measuring the CPU time for the current thread.
+ */
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_CURRENT_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isCurrentThreadCpuTimeSupported();
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+ private static boolean IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED = false;
+ private long _startTimeNs = -1;
+ private long _endTimeNs = -1;
+
+ public ThreadTimer() {
+ }
+
+ public static void setThreadCpuTimeMeasurementEnabled(boolean enable) {
+ IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED = enable && IS_CURRENT_THREAD_CPU_TIME_SUPPORTED;
+ }
+
+ public void start() {
+ if (IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED) {
+ _startTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+ }
+ }
+
+ public void stop() {
+ if (IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED) {
+ _endTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+ }
+ }
+
+ public long getThreadTimeNs() {
+ if (_startTimeNs == -1 || _endTimeNs == -1) {
Review comment:
Will send another PR to address this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-797837587
We should emit the metric like other server metrics and also see if the measurement can be made configurable via instance level config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595676976
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/InstanceResponseBlock.java
##########
@@ -41,6 +41,9 @@
public InstanceResponseBlock(IntermediateResultsBlock intermediateResultsBlock) {
try {
_instanceResponseDataTable = intermediateResultsBlock.getDataTable();
+ long totalThreadCpuTimeNs = intermediateResultsBlock.getThreadCpuTimeNs();
+ _instanceResponseDataTable.getMetadata().put(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY,
Review comment:
I think logging should still be done in QueryScheduler since that's where all the metrics of a query are logged together in one line once the processing is done. So, we shouldn't log it in a new place now imo.
As you suggested, I think we can do a put() here, retrieve the metric in QueryScheduler from dataTable metadata and then make sure to do a remove() before `byte[] responseData = serializeDataTable(queryRequest, dataTable);` is called. Use the retrieved value for adding to table gauge and writing the log. For the remove(), add a TODO to revisit this when follow-up work of data table serialization cost measurement is done. Should be fine since we are going to do the follow-up immediately after this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r597095435
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/operator/combine/CombineSlowOperatorsTest.java
##########
@@ -65,7 +65,8 @@ public void setUp() {
public void testSelectionOnlyCombineOperator() {
List<Operator> operators = getOperators();
SelectionOnlyCombineOperator combineOperator = new SelectionOnlyCombineOperator(operators,
- QueryContextConverterUtils.getQueryContextFromPQL("SELECT * FROM table"), _executorService, TIMEOUT_MS);
+ QueryContextConverterUtils.getQueryContextFromPQL("SELECT * FROM table"),
Review comment:
Oh, this is performed by IntellJ reformat during rebase. Since it's a common practice to limit the length of line and this PR already get merged, I vote to not undo this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r593733086
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isThreadCpuTimeSupported();
+ private static final long NANOS_IN_MILLIS = 1;
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+
+ private long _startTimeMs = -1;
+ private long _endTimeMs = -1;
+
+ public ThreadTimer() {
+ }
+
+ public void start() {
+ if (IS_THREAD_CPU_TIME_SUPPORTED) {
+ _startTimeMs = MX_BEAN.getCurrentThreadCpuTime() / NANOS_IN_MILLIS;
+ }
+ }
+
+ public void stop() {
+ if (IS_THREAD_CPU_TIME_SUPPORTED) {
+ _endTimeMs = MX_BEAN.getCurrentThreadCpuTime() / NANOS_IN_MILLIS;
+ }
+ }
+
+ public long getThreadTime() {
+ if (_startTimeMs == -1 || _endTimeMs == -1) {
+ return 0;
+ }
+ return _endTimeMs - _startTimeMs;
+ }
+
+ static {
+ LOGGER.info("Thread cpu time measurement supported: {}", IS_THREAD_CPU_TIME_SUPPORTED);
+ }
+}
Review comment:
nit: newline
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang closed pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang closed pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r594783322
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-797826488
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=h1) Report
> Merging [#6680](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=desc) (bbeb74a) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `0.60%`.
> The diff coverage is `62.64%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6680 +/- ##
==========================================
- Coverage 66.44% 65.84% -0.61%
==========================================
Files 1075 1389 +314
Lines 54773 67525 +12752
Branches 8168 9789 +1621
==========================================
+ Hits 36396 44464 +8068
- Misses 15700 19899 +4199
- Partials 2677 3162 +485
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.84% <62.64%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...pinot/broker/api/resources/PinotClientRequest.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdENsaWVudFJlcXVlc3QuamF2YQ==) | `0.00% <0.00%> (-27.28%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <ø> (+9.52%)` | :arrow_up: |
| [...n/java/org/apache/pinot/client/BrokerResponse.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Jyb2tlclJlc3BvbnNlLmphdmE=) | `100.00% <ø> (ø)` | |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <ø> (-13.29%)` | :arrow_down: |
| [...org/apache/pinot/client/DynamicBrokerSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0R5bmFtaWNCcm9rZXJTZWxlY3Rvci5qYXZh) | `82.85% <ø> (+10.12%)` | :arrow_up: |
| [...n/java/org/apache/pinot/client/ExecutionStats.java](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0V4ZWN1dGlvblN0YXRzLmphdmE=) | `68.88% <ø> (ø)` | |
| ... and [1274 more](https://codecov.io/gh/apache/incubator-pinot/pull/6680/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=footer). Last update [656f279...bbeb74a](https://codecov.io/gh/apache/incubator-pinot/pull/6680?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-800831123
rebased and all comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595594826
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
##########
@@ -317,6 +317,9 @@
public static final String ACCESS_CONTROL_FACTORY_CLASS = "pinot.server.admin.access.control.factory.class";
public static final String DEFAULT_ACCESS_CONTROL_FACTORY_CLASS =
"org.apache.pinot.server.api.access.AllowAllAccessFactory";
+
+ public static final String CONFIG_OF_ENABLE_THREAD_CPU_TIME_INSTRUMENT = "pinot.server.instance.enableThreadCpuTimeInstrument";
+ public static final boolean DEFAULT_ENABLE_THREAD_CPU_TIME_INSTRUMENT = true;
Review comment:
default should be false, at least for now
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java
##########
@@ -65,6 +65,7 @@
SEGMENT_DOWNLOAD_FAILURES("segments", false),
NUM_RESIZES("numResizes", false),
RESIZE_TIME_MS("resizeTimeMs", false),
+ THREAD_CPU_TIME_NS("threadCpuTimeNs", false),
Review comment:
should be in `ServerGauge` not `ServerMeter`. Also looks like `RESIZE_TIME` is in the wrong place. Should be a gauge. You can fix that in a separate PR
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/InstanceResponseBlock.java
##########
@@ -41,6 +41,9 @@
public InstanceResponseBlock(IntermediateResultsBlock intermediateResultsBlock) {
try {
_instanceResponseDataTable = intermediateResultsBlock.getDataTable();
+ long totalThreadCpuTimeNs = intermediateResultsBlock.getThreadCpuTimeNs();
+ _instanceResponseDataTable.getMetadata().put(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY,
Review comment:
Do we want to hold off on this until we also add the serialization part?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r597094046
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java
##########
@@ -77,6 +77,7 @@
private PlanMaker _planMaker;
private long _defaultTimeOutMs = CommonConstants.Server.DEFAULT_QUERY_EXECUTOR_TIMEOUT_MS;
private ServerMetrics _serverMetrics;
+ boolean _enableThreadCpuTimeInstrument;
Review comment:
Will send another PR to address this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r593790598
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/IntermediateResultsBlock.java
##########
@@ -69,6 +69,7 @@
private boolean _numGroupsLimitReached;
private int _numResizes;
private long _resizeTimeMs;
+ private long _threadTime;
Review comment:
suggest: `threadCpuTimeMicros` instead of `threadTime` (Hint: Always add the units in variable name)
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isThreadCpuTimeSupported();
+ private static final long NANOS_IN_MILLIS = 1;
Review comment:
Make it MICROS_IN_MILLIS
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -53,15 +55,16 @@
protected final long _endTimeMs;
protected final int _numOperators;
- protected int _numThreads;
// Use a Phaser to ensure all the Futures are done (not scheduled, finished or interrupted) before the main thread
// returns. We need to ensure this because the main thread holds the reference to the segments. If a segment is
// deleted/refreshed, the segment will be released after the main thread returns, which would lead to undefined
// behavior (even JVM crash) when processing queries against it.
protected final Phaser _phaser = new Phaser(1);
- protected Future[] _futures;
// Use a _blockingQueue to store the per-segment result
private final BlockingQueue<IntermediateResultsBlock> _blockingQueue;
+ private final AtomicLong totalWorkerTime = new AtomicLong(0);
Review comment:
Suggest: `totalWorkerThreadCpuTimeMicros` or `totalThreadCpuMicros` or `execCpuTimeMicros` . Definitely include units
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -89,12 +92,28 @@ protected IntermediateResultsBlock getNextBlock() {
_futures[i] = _executorService.submit(new TraceRunnable() {
@Override
public void runJob() {
+ ThreadTimer processThreadTimer = new ThreadTimer();
Review comment:
Suggest `executionThreadTimer`
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java
##########
@@ -44,6 +44,7 @@
String REQUEST_ID_METADATA_KEY = "requestId";
String NUM_RESIZES_METADATA_KEY = "numResizes";
String RESIZE_TIME_MS_METADATA_KEY = "resizeTimeMs";
+ String THREAD_TIME_MS = "threadTimeMs";
Review comment:
Change it to micro seconds please
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isThreadCpuTimeSupported();
+ private static final long NANOS_IN_MILLIS = 1;
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+
+ private long _startTimeMs = -1;
Review comment:
Also introduce a static method that detects whether the underlying JVM supports getCurrentThreadCpuTme. If that is not supported (or if thread cpu time is not configured), then the overhead should be close to nothing.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isThreadCpuTimeSupported();
+ private static final long NANOS_IN_MILLIS = 1;
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+
+ private long _startTimeMs = -1;
Review comment:
We need microseconds, not milliseconds
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
Review comment:
Make this class configurable via instance configuration. If the instance configuration asks not to measure this, then do as appropriate in in the class below, with as minimal an overhead as possible. Even though we have measured the overhead to be near-zero in high throughput use cases, it is always useful to have a mechanism to turn it off in production when we first introduce it. It will also help us roll out the feature slowly in specific clusters or data centers
Eventually, we will get rid of the config and make it standard.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/streaming/StreamingSelectionOnlyCombineOperator.java
##########
@@ -47,6 +47,7 @@
/**
* Combine operator for selection only streaming queries.
+ * TODO: extends StreamingSelectionOnlyCombineOperator from BaseCombineOperator.
Review comment:
```suggestion
* TODO: extend StreamingSelectionOnlyCombineOperator from BaseCombineOperator.
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
Review comment:
+1
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -89,12 +92,28 @@ protected IntermediateResultsBlock getNextBlock() {
_futures[i] = _executorService.submit(new TraceRunnable() {
@Override
public void runJob() {
+ ThreadTimer processThreadTimer = new ThreadTimer();
+ processThreadTimer.start();
+
processSegments(threadIndex);
+
+ processThreadTimer.stop();
+ totalWorkerTime.addAndGet(processThreadTimer.getThreadTime());
Review comment:
Maybe we can introduce another API `ThreadTimer.stopAndGet()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596203539
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,76 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code ThreadTimer} class providing the functionality of measuring the CPU time for the current thread.
+ */
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_CURRENT_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isCurrentThreadCpuTimeSupported();
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+ private static boolean IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED;
+ private long _startTimeNs = -1;
+ private long _endTimeNs = -1;
+
+ public ThreadTimer() {
+ }
+
+ public static boolean isCurrentThreadCpuTimeSupported() {
Review comment:
remove this method, we should not need it.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,76 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code ThreadTimer} class providing the functionality of measuring the CPU time for the current thread.
+ */
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_CURRENT_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isCurrentThreadCpuTimeSupported();
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+ private static boolean IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED;
+ private long _startTimeNs = -1;
+ private long _endTimeNs = -1;
+
+ public ThreadTimer() {
+ }
+
+ public static boolean isCurrentThreadCpuTimeSupported() {
+ return IS_CURRENT_THREAD_CPU_TIME_SUPPORTED;
+ }
+
+ public void start() {
+ if (IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED && IS_CURRENT_THREAD_CPU_TIME_SUPPORTED) {
+ _startTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+ }
+ }
+
+ public void stop() {
+ if (IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED && IS_CURRENT_THREAD_CPU_TIME_SUPPORTED) {
+ _endTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+ }
+ }
+
+ public long getThreadTimeNs() {
+ if (_startTimeNs == -1 || _endTimeNs == -1) {
+ return 0;
+ }
+ return _endTimeNs - _startTimeNs;
+ }
+
+ public long stopAndGetThreadTimeNs() {
+ stop();
+ return getThreadTimeNs();
+ }
+
+ static {
+ LOGGER.info("Current thread cpu time measurement supported: {}", IS_CURRENT_THREAD_CPU_TIME_SUPPORTED);
+ }
+
+ public static void setThreadCpuTimeMeasurementEnabled(boolean enable) {
+ IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED = enable;
Review comment:
In this method, call ManagementFactory.getMxBean, and then decide once and for all whether we are going to measure thread cpu time or not, and store it in a single boolean variable, to be checked every time start/stop is invoked.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r593733021
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isThreadCpuTimeSupported();
+ private static final long NANOS_IN_MILLIS = 1;
Review comment:
this should be 10^6?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r594783420
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isThreadCpuTimeSupported();
+ private static final long NANOS_IN_MILLIS = 1;
Review comment:
done
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isThreadCpuTimeSupported();
+ private static final long NANOS_IN_MILLIS = 1;
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+
+ private long _startTimeMs = -1;
+ private long _endTimeMs = -1;
+
+ public ThreadTimer() {
+ }
+
+ public void start() {
+ if (IS_THREAD_CPU_TIME_SUPPORTED) {
+ _startTimeMs = MX_BEAN.getCurrentThreadCpuTime() / NANOS_IN_MILLIS;
+ }
+ }
+
+ public void stop() {
+ if (IS_THREAD_CPU_TIME_SUPPORTED) {
+ _endTimeMs = MX_BEAN.getCurrentThreadCpuTime() / NANOS_IN_MILLIS;
+ }
+ }
+
+ public long getThreadTime() {
+ if (_startTimeMs == -1 || _endTimeMs == -1) {
+ return 0;
+ }
+ return _endTimeMs - _startTimeMs;
+ }
+
+ static {
+ LOGGER.info("Thread cpu time measurement supported: {}", IS_THREAD_CPU_TIME_SUPPORTED);
+ }
+}
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r597032543
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code ThreadTimer} class providing the functionality of measuring the CPU time for the current thread.
+ */
+public class ThreadTimer {
+ private static final ThreadMXBean MX_BEAN = ManagementFactory.getThreadMXBean();
+ private static final boolean IS_CURRENT_THREAD_CPU_TIME_SUPPORTED = MX_BEAN.isCurrentThreadCpuTimeSupported();
+ private static final Logger LOGGER = LoggerFactory.getLogger(ThreadTimer.class);
+ private static boolean IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED = false;
+ private long _startTimeNs = -1;
+ private long _endTimeNs = -1;
+
+ public ThreadTimer() {
+ }
+
+ public static void setThreadCpuTimeMeasurementEnabled(boolean enable) {
+ IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED = enable && IS_CURRENT_THREAD_CPU_TIME_SUPPORTED;
+ }
+
+ public void start() {
+ if (IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED) {
+ _startTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+ }
+ }
+
+ public void stop() {
+ if (IS_THREAD_CPU_TIME_MEASUREMENT_ENABLED) {
+ _endTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+ }
+ }
+
+ public long getThreadTimeNs() {
+ if (_startTimeNs == -1 || _endTimeNs == -1) {
Review comment:
Do we need this if check? What is the worst that can happen if we call `get` without calling start? I vote for avoiding this. Instead if thread cpu timing is not enabled, return 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-798779416
Just to clarify:
The value returned is in nanos. Initially in the prototype I had this converted to milli seconds, and most values turned out to be 0 when dividing nanos by 1_000_000 to get millis. So I changed it to log the values in nanos, and found that the resolution was much better (i;.e. we get non-zero numbers). It is ok to convert to microseconds. Or just leave it at nanoseconds. But converting to millis will not work
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r594783534
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -53,15 +55,16 @@
protected final long _endTimeMs;
protected final int _numOperators;
- protected int _numThreads;
// Use a Phaser to ensure all the Futures are done (not scheduled, finished or interrupted) before the main thread
// returns. We need to ensure this because the main thread holds the reference to the segments. If a segment is
// deleted/refreshed, the segment will be released after the main thread returns, which would lead to undefined
// behavior (even JVM crash) when processing queries against it.
protected final Phaser _phaser = new Phaser(1);
- protected Future[] _futures;
// Use a _blockingQueue to store the per-segment result
private final BlockingQueue<IntermediateResultsBlock> _blockingQueue;
+ private final AtomicLong totalWorkerTime = new AtomicLong(0);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r593511712
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/ThreadTimer.java
##########
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.request.context;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ThreadTimer {
Review comment:
Please add javadoc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596451009
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java
##########
@@ -41,7 +41,7 @@
REALTIME_OFFHEAP_MEMORY_USED("bytes", false),
REALTIME_SEGMENT_NUM_PARTITIONS("realtimeSegmentNumPartitions", false),
LLC_SIMULTANEOUS_SEGMENT_BUILDS("llcSimultaneousSegmentBuilds", true),
-
+ EXECUTION_THREAD_CPU_TIME_NS("executionThreadCpuTimeNs", false),
Review comment:
We should pass the unit as the first argument.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595615814
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -89,12 +92,28 @@ protected IntermediateResultsBlock getNextBlock() {
_futures[i] = _executorService.submit(new TraceRunnable() {
@Override
public void runJob() {
+ ThreadTimer processThreadTimer = new ThreadTimer();
+ processThreadTimer.start();
+
processSegments(threadIndex);
+
+ processThreadTimer.stop();
+ totalWorkerTime.addAndGet(processThreadTimer.getThreadTime());
}
});
}
+ ThreadTimer mergeThreadTimer = new ThreadTimer();
Review comment:
As discussed offline, we should move all the single threaded instrumentation to InstanceResponseOperator
```
@Override
protected InstanceResponseBlock getNextBlock() {
ThreadTimer mainThreadTimer = new ThreadTimer();
mainThreadTimer.start();
IntermediateResultsBlock intermediateResultsBlock = (IntermediateResultsBlock) _operator.nextBlock();
InstanceResponseBlock instanceResponseBlock = new InstanceResponseBlock(intermediateResultsBlock);
DataTable dataTable = instanceResponseBlock.getInstanceResponseDataTable();
mainThreadTimer.stop();
long totalThreadCpuTimeNs = intermediateResultsBlock.getThreadCpuTimeNs() + mainThreadTimer.getThreadTimeNs();
dataTable.getMetadata().put(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY, String.valueOf(totalThreadCpuTimeNs));
return instanceResponseBlock
}
```
This way all single threaded query execution (except for planning) code's cpu time is covered without having to do it in multiple places.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595677362
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java
##########
@@ -65,6 +65,7 @@
SEGMENT_DOWNLOAD_FAILURES("segments", false),
NUM_RESIZES("numResizes", false),
RESIZE_TIME_MS("resizeTimeMs", false),
+ THREAD_CPU_TIME_NS("threadCpuTimeNs", false),
Review comment:
@mqliang ^^, let's change the metric name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r594777195
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -89,12 +92,28 @@ protected IntermediateResultsBlock getNextBlock() {
_futures[i] = _executorService.submit(new TraceRunnable() {
@Override
public void runJob() {
+ ThreadTimer processThreadTimer = new ThreadTimer();
+ processThreadTimer.start();
+
processSegments(threadIndex);
+
+ processThreadTimer.stop();
+ totalWorkerTime.addAndGet(processThreadTimer.getThreadTime());
}
});
}
+ ThreadTimer mergeThreadTimer = new ThreadTimer();
Review comment:
If we instrument in that way (start the main thread timer in InstanceResponseOperator), then the total time will account an extra processing thread time interval:
example:
t0: InstanceResponseOperator call InstanceResponseBlock's constructor, which will call _operator.nextBlock();
t2: process segments in parallel. let's say we have 5 thread processing segments in parallel. Let's assume all the 5 thread start and end at the same time.
t3: processing end, start merge
t4: merge end.
NOTE: since time spending on constructor is negligible, let's assume t0=t2.
Then, expected_totalTime = processing_time + merge_time = (t3-t2)*5 + (t4-t3). However, if start the main thread timer in InstanceResponseOperator, actual_totalTime = t4-t0+(t3-t2)*5 = t4-t2+(t3-t2)*5
actual_totalTime - expected_totalTime = (t4-t2)-(t4-t3) = t3-t2, in other word, there is an extra processing thread time get accounted.
I will let @mcvsubbu chime in to make a double check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
mqliang commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-801577577
conflict with master since https://github.com/apache/incubator-pinot/pull/6690 get merged. Rebased and updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators query execution code with thread cpu time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596478073
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java
##########
@@ -41,7 +41,7 @@
REALTIME_OFFHEAP_MEMORY_USED("bytes", false),
REALTIME_SEGMENT_NUM_PARTITIONS("realtimeSegmentNumPartitions", false),
LLC_SIMULTANEOUS_SEGMENT_BUILDS("llcSimultaneousSegmentBuilds", true),
-
+ EXECUTION_THREAD_CPU_TIME_NS("executionThreadCpuTimeNs", false),
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r596447139
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/operator/combine/CombineSlowOperatorsTest.java
##########
@@ -65,7 +65,8 @@ public void setUp() {
public void testSelectionOnlyCombineOperator() {
List<Operator> operators = getOperators();
SelectionOnlyCombineOperator combineOperator = new SelectionOnlyCombineOperator(operators,
- QueryContextConverterUtils.getQueryContextFromPQL("SELECT * FROM table"), _executorService, TIMEOUT_MS);
+ QueryContextConverterUtils.getQueryContextFromPQL("SELECT * FROM table"),
Review comment:
This change doesn't seem to be related to the PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595650931
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java
##########
@@ -65,6 +65,7 @@
SEGMENT_DOWNLOAD_FAILURES("segments", false),
NUM_RESIZES("numResizes", false),
RESIZE_TIME_MS("resizeTimeMs", false),
+ THREAD_CPU_TIME_NS("threadCpuTimeNs", false),
Review comment:
Agreed with changing the name to execution thread time.
I think the phase time just comes out as a gauge, so it is the same thing. Since this is not a separate query phase we are monitoring, I would leave it as a gauge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-797838599
> We should emit the metric like other server metrics and also see if the measurement can be made configurable via instance level config.
This is a query level metric. If we emit it as BrokerMetrics/BrokerGauge, the number of metrics will explode. But we can expose it as a part of query response metadata, just like numServersQueried/numDocsScanned).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mqliang commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595642761
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/InstanceResponseBlock.java
##########
@@ -41,6 +41,9 @@
public InstanceResponseBlock(IntermediateResultsBlock intermediateResultsBlock) {
try {
_instanceResponseDataTable = intermediateResultsBlock.getDataTable();
+ long totalThreadCpuTimeNs = intermediateResultsBlock.getThreadCpuTimeNs();
+ _instanceResponseDataTable.getMetadata().put(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY,
Review comment:
If we don't set the value as DataTable metadata here, then it's impossible to get the value in QueryScheduler.java, in other words, we should delete the ServerGauge emitting code also. I think at this moment, we can just put a log line here, and deleting the change in QueryScheduler.java.
Another workaround is: add the value as DataTable metadata here, but in QueryScheduler.java, after writing log and emitting ServerGauge, call`dataTable.getMetadata().remove(DataTable.THREAD_CPU_TIME_NS_METADATA_KEY)` so that this value will NOT be send back to broker (and add a TODO stating remove it when we have data table serialization cost measuring).
Any preference between the two option?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#discussion_r595643785
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java
##########
@@ -65,6 +65,7 @@
SEGMENT_DOWNLOAD_FAILURES("segments", false),
NUM_RESIZES("numResizes", false),
RESIZE_TIME_MS("resizeTimeMs", false),
+ THREAD_CPU_TIME_NS("threadCpuTimeNs", false),
Review comment:
Thinking more about this
We have ServerMeter, ServerGauge, ServerTimer and ServerQueryPhase.
- This metric doesn't fit into ServerQueryPhase since we are not introducing a new phase.
- Similarly, ServerMeter is also not right as it is not used for emitting duration related metrics.
So between ServerGauge and ServerTimer, I am thinking we should use ServerTimer. Currently it is used for freshness lag and netty server response latency. We can introduce a new timer for CPU time measurements and then use `serverMetrics.addTimedTableValue(tableNameWithType, ServerTimer.THREAD_CPU_TIME ....)`
Now for the naming, we currently use `THREAD_CPU_TIME_NS`. Right now our focus is on two main costs for query execution
- Execution for segment level and combine - this PR
- DataTable serialization - next PR
For sending to the broker, it can add up both, but I feel we should have different metric names for cpu timers for the server to emit both separately. So, something like EXECUTION_THREAD_CPU_TIME_NS and RESPONSE_SERIALIZATION_CPU_TIME_NS.
Secondly, in future we might decide to emit the metrics for let's say pruning, physical planning etc. Server can continue to add up everything and send to broker, but I think it should still emit individually for which we need separate names
@mcvsubbu @mqliang what do you think ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on pull request #6680: Instrument combine operators with total thread time
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on pull request #6680:
URL: https://github.com/apache/incubator-pinot/pull/6680#issuecomment-800820919
@mqliang , can you please rebase this PR since https://github.com/apache/incubator-pinot/pull/6672 has been merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org