You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/04/14 00:02:34 UTC
[GitHub] [pinot] npawar opened a new pull request, #8539: Add brokerURL and better handle TimeoutException in QueryRunner
npawar opened a new pull request, #8539:
URL: https://github.com/apache/pinot/pull/8539
Add support for QueryRunner with brokerURL
Don't throw exception if reached configured timeout, instead proceed as normal to print final statictics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r850688941
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -416,42 +422,49 @@ public static QuerySummary multiThreadedQueryRunner(PerfBenchmarkDriverConf conf
long reportStartTime = startTime;
int numReportIntervals = 0;
int numTimesExecuted = 0;
- while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
- if (executorService.isTerminated()) {
- throw new IllegalThreadStateException("All threads got exception and already dead.");
- }
-
- if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
- throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
- }
+ try {
+ while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
+ if (executorService.isTerminated()) {
+ throw new IllegalThreadStateException("All threads got exception and already dead.");
+ }
- for (String query : queries) {
- while (!queryQueue.offer(query)) {
- Thread.sleep(1);
+ if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
+ throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
}
- long currentTime = System.currentTimeMillis();
- if (currentTime - reportStartTime >= reportIntervalMs) {
- long timePassed = currentTime - startTime;
- int numQueriesExecutedInt = numQueriesExecuted.get();
- LOGGER.info("Time Passed: {}ms, Queries Executed: {}, Exceptions: {}, Average QPS: {}, "
- + "Average Broker Time: {}ms, Average Client Time: {}ms.", timePassed, numQueriesExecutedInt,
- numExceptions.get(), numQueriesExecutedInt / ((double) timePassed / MILLIS_PER_SECOND),
- totalBrokerTime.get() / (double) numQueriesExecutedInt,
- totalClientTime.get() / (double) numQueriesExecutedInt);
- reportStartTime = currentTime;
- numReportIntervals++;
-
- if ((numIntervalsToReportAndClearStatistics != 0) && (numReportIntervals
- == numIntervalsToReportAndClearStatistics)) {
- numReportIntervals = 0;
- startTime = currentTime;
- reportAndClearStatistics(numQueriesExecuted, numExceptions, totalBrokerTime, totalClientTime,
- statisticsList);
+ for (String query : queries) {
+ if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
+ throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
+ }
+ while (!queryQueue.offer(query)) {
+ Thread.sleep(1);
+ }
+
+ long currentTime = System.currentTimeMillis();
+ if (currentTime - reportStartTime >= reportIntervalMs) {
+ long timePassed = currentTime - startTime;
+ int numQueriesExecutedInt = numQueriesExecuted.get();
+ LOGGER.info("Time Passed: {}ms, Queries Executed: {}, Exceptions: {}, Average QPS: {}, " + "Average "
+ + "Broker Time: {}ms, Average Client Time: {}ms.", timePassed, numQueriesExecutedInt,
+ numExceptions.get(), numQueriesExecutedInt / ((double) timePassed / MILLIS_PER_SECOND),
+ totalBrokerTime.get() / (double) numQueriesExecutedInt,
+ totalClientTime.get() / (double) numQueriesExecutedInt);
+ reportStartTime = currentTime;
+ numReportIntervals++;
+
+ if ((numIntervalsToReportAndClearStatistics != 0) && (numReportIntervals
+ == numIntervalsToReportAndClearStatistics)) {
+ numReportIntervals = 0;
+ startTime = currentTime;
+ reportAndClearStatistics(numQueriesExecuted, numExceptions, totalBrokerTime, totalClientTime,
+ statisticsList);
+ }
}
}
+ numTimesExecuted++;
}
- numTimesExecuted++;
+ } catch (TimeoutException e) {
+ LOGGER.info("Time out reached. {}", e.getMessage());
Review Comment:
It's not. Event after this change, you'll still get the message `Time out reached. Timeout of 300000 sec reached. Aborting`
Prior to this change, after this an exception was thrown and no final statistics were printed.
After this change, we stop gracefully ad print the final stats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r849984702
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -299,50 +302,53 @@ public static QuerySummary singleThreadedQueryRunner(PerfBenchmarkDriverConf con
long reportStartTime = startTime;
int numReportIntervals = 0;
int numTimesExecuted = 0;
- while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
- for (String query : queries) {
- if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
- LOGGER.warn("Timeout of {} sec reached. Aborting", timeout);
- throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
- }
+ try {
Review Comment:
there's no change here other than adding TimeoutException and catching it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r852320632
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -53,56 +52,58 @@ public class QueryRunner extends AbstractBaseCommand implements Command {
private static final long NANO_DELTA = (long) 5E5;
private static final String CLIENT_TIME_STATISTICS = "CLIENT TIME STATISTICS";
- @CommandLine.Option(names = {"-mode"}, required = true,
- description = "Mode of query runner (singleThread|multiThreads|targetQPS|increasingQPS).")
+ @CommandLine.Option(names = {"-mode"}, required = true, description = "Mode of query runner "
+ + "(singleThread|multiThreads|targetQPS|increasingQPS).")
Review Comment:
These changes do not seem to improve readability. IIRC we have disabled these kind of auto-reformat in the `Pinot Style`, can you try re-importing the style sheet?
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -295,15 +297,18 @@ public static QuerySummary singleThreadedQueryRunner(PerfBenchmarkDriverConf con
List<Statistics> statisticsList = Collections.singletonList(new Statistics(CLIENT_TIME_STATISTICS));
final long startTimeAbsolute = System.currentTimeMillis();
+ boolean timeoutReached = false;
long startTime = System.currentTimeMillis();
long reportStartTime = startTime;
int numReportIntervals = 0;
int numTimesExecuted = 0;
- while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
+
+ while (!timeoutReached && (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries)) {
for (String query : queries) {
if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
- LOGGER.warn("Timeout of {} sec reached. Aborting", timeout);
- throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
+ LOGGER.info("Timeout of {} sec reached. Aborting", timeout);
Review Comment:
Suggest keeping it as warning. Same for other places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r852433622
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -295,15 +297,18 @@ public static QuerySummary singleThreadedQueryRunner(PerfBenchmarkDriverConf con
List<Statistics> statisticsList = Collections.singletonList(new Statistics(CLIENT_TIME_STATISTICS));
final long startTimeAbsolute = System.currentTimeMillis();
+ boolean timeoutReached = false;
long startTime = System.currentTimeMillis();
long reportStartTime = startTime;
int numReportIntervals = 0;
int numTimesExecuted = 0;
- while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
+
+ while (!timeoutReached && (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries)) {
for (String query : queries) {
if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
- LOGGER.warn("Timeout of {} sec reached. Aborting", timeout);
- throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
+ LOGGER.info("Timeout of {} sec reached. Aborting", timeout);
Review Comment:
imo this should be info. It is doing exactly what the config is asking it to do, and it ending here isn't an erroneous condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r853554426
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -53,56 +52,58 @@ public class QueryRunner extends AbstractBaseCommand implements Command {
private static final long NANO_DELTA = (long) 5E5;
private static final String CLIENT_TIME_STATISTICS = "CLIENT TIME STATISTICS";
- @CommandLine.Option(names = {"-mode"}, required = true,
- description = "Mode of query runner (singleThread|multiThreads|targetQPS|increasingQPS).")
+ @CommandLine.Option(names = {"-mode"}, required = true, description = "Mode of query runner "
+ + "(singleThread|multiThreads|targetQPS|increasingQPS).")
Review Comment:
ah i see! will keep in mind for next time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8539:
URL: https://github.com/apache/pinot/pull/8539#issuecomment-1098603629
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8539](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b1f25ce) into [master](https://codecov.io/gh/apache/pinot/commit/476679da861f20b7c7b1100f818a03eae60a9776?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (476679d) will **increase** coverage by `41.43%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #8539 +/- ##
=============================================
+ Coverage 29.10% 70.53% +41.43%
- Complexity 0 4290 +4290
=============================================
Files 1669 1681 +12
Lines 87585 87949 +364
Branches 13274 13317 +43
=============================================
+ Hits 25491 62038 +36547
+ Misses 59770 21601 -38169
- Partials 2324 4310 +1986
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `26.87% <ø> (+0.04%)` | :arrow_up: |
| integration2 | `25.90% <ø> (+0.08%)` | :arrow_up: |
| unittests1 | `67.02% <ø> (?)` | |
| unittests2 | `14.06% <ø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | `71.42% <0.00%> (-1.43%)` | :arrow_down: |
| [...pache/pinot/core/query/utils/idset/EmptyIdSet.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS91dGlscy9pZHNldC9FbXB0eUlkU2V0LmphdmE=) | `25.00% <0.00%> (ø)` | |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `68.27% <0.00%> (ø)` | |
| [...anager/realtime/SegmentBuildTimeLeaseExtender.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VnbWVudEJ1aWxkVGltZUxlYXNlRXh0ZW5kZXIuamF2YQ==) | `63.23% <0.00%> (ø)` | |
| [...in/stream/kafka20/KafkaPartitionLevelConsumer.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL0thZmthUGFydGl0aW9uTGV2ZWxDb25zdW1lci5qYXZh) | `66.66% <0.00%> (ø)` | |
| [...pache/pinot/ingestion/jobs/SegmentCreationJob.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vam9icy9TZWdtZW50Q3JlYXRpb25Kb2IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [.../pinot/ingestion/jobs/SegmentPreprocessingJob.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vam9icy9TZWdtZW50UHJlcHJvY2Vzc2luZ0pvYi5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...api/resources/ControllerEchoWithAutoDiscovery.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtaW50ZWdyYXRpb24tdGVzdHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L2NvbnRyb2xsZXIvYXBpL3Jlc291cmNlcy9Db250cm9sbGVyRWNob1dpdGhBdXRvRGlzY292ZXJ5LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/pinot/ingestion/jobs/SegmentTarPushJob.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vam9icy9TZWdtZW50VGFyUHVzaEpvYi5qYXZh) | `70.27% <0.00%> (ø)` | |
| [...ker/api/resources/BrokerEchoWithAutoDiscovery.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtaW50ZWdyYXRpb24tdGVzdHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L2Jyb2tlci9hcGkvcmVzb3VyY2VzL0Jyb2tlckVjaG9XaXRoQXV0b0Rpc2NvdmVyeS5qYXZh) | `0.00% <0.00%> (ø)` | |
| ... and [1164 more](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [476679d...b1f25ce](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar merged pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
npawar merged PR #8539:
URL: https://github.com/apache/pinot/pull/8539
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
richardstartin commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r850312126
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -416,42 +422,49 @@ public static QuerySummary multiThreadedQueryRunner(PerfBenchmarkDriverConf conf
long reportStartTime = startTime;
int numReportIntervals = 0;
int numTimesExecuted = 0;
- while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
- if (executorService.isTerminated()) {
- throw new IllegalThreadStateException("All threads got exception and already dead.");
- }
-
- if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
- throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
- }
+ try {
+ while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
+ if (executorService.isTerminated()) {
+ throw new IllegalThreadStateException("All threads got exception and already dead.");
+ }
- for (String query : queries) {
- while (!queryQueue.offer(query)) {
- Thread.sleep(1);
+ if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
+ throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
}
- long currentTime = System.currentTimeMillis();
- if (currentTime - reportStartTime >= reportIntervalMs) {
- long timePassed = currentTime - startTime;
- int numQueriesExecutedInt = numQueriesExecuted.get();
- LOGGER.info("Time Passed: {}ms, Queries Executed: {}, Exceptions: {}, Average QPS: {}, "
- + "Average Broker Time: {}ms, Average Client Time: {}ms.", timePassed, numQueriesExecutedInt,
- numExceptions.get(), numQueriesExecutedInt / ((double) timePassed / MILLIS_PER_SECOND),
- totalBrokerTime.get() / (double) numQueriesExecutedInt,
- totalClientTime.get() / (double) numQueriesExecutedInt);
- reportStartTime = currentTime;
- numReportIntervals++;
-
- if ((numIntervalsToReportAndClearStatistics != 0) && (numReportIntervals
- == numIntervalsToReportAndClearStatistics)) {
- numReportIntervals = 0;
- startTime = currentTime;
- reportAndClearStatistics(numQueriesExecuted, numExceptions, totalBrokerTime, totalClientTime,
- statisticsList);
+ for (String query : queries) {
+ if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
+ throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
+ }
+ while (!queryQueue.offer(query)) {
+ Thread.sleep(1);
+ }
+
+ long currentTime = System.currentTimeMillis();
+ if (currentTime - reportStartTime >= reportIntervalMs) {
+ long timePassed = currentTime - startTime;
+ int numQueriesExecutedInt = numQueriesExecuted.get();
+ LOGGER.info("Time Passed: {}ms, Queries Executed: {}, Exceptions: {}, Average QPS: {}, " + "Average "
+ + "Broker Time: {}ms, Average Client Time: {}ms.", timePassed, numQueriesExecutedInt,
+ numExceptions.get(), numQueriesExecutedInt / ((double) timePassed / MILLIS_PER_SECOND),
+ totalBrokerTime.get() / (double) numQueriesExecutedInt,
+ totalClientTime.get() / (double) numQueriesExecutedInt);
+ reportStartTime = currentTime;
+ numReportIntervals++;
+
+ if ((numIntervalsToReportAndClearStatistics != 0) && (numReportIntervals
+ == numIntervalsToReportAndClearStatistics)) {
+ numReportIntervals = 0;
+ startTime = currentTime;
+ reportAndClearStatistics(numQueriesExecuted, numExceptions, totalBrokerTime, totalClientTime,
+ statisticsList);
+ }
}
}
+ numTimesExecuted++;
}
- numTimesExecuted++;
+ } catch (TimeoutException e) {
+ LOGGER.info("Time out reached. {}", e.getMessage());
Review Comment:
timeouts should not be omitted from the reported statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r853540172
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -53,56 +52,58 @@ public class QueryRunner extends AbstractBaseCommand implements Command {
private static final long NANO_DELTA = (long) 5E5;
private static final String CLIENT_TIME_STATISTICS = "CLIENT TIME STATISTICS";
- @CommandLine.Option(names = {"-mode"}, required = true,
- description = "Mode of query runner (singleThread|multiThreads|targetQPS|increasingQPS).")
+ @CommandLine.Option(names = {"-mode"}, required = true, description = "Mode of query runner "
+ + "(singleThread|multiThreads|targetQPS|increasingQPS).")
Review Comment:
The new one will honor the new lines and not try to merge separate lines, so if you revert and reformat, it won't apply the same change here. Not important though :-P
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r852432915
##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -53,56 +52,58 @@ public class QueryRunner extends AbstractBaseCommand implements Command {
private static final long NANO_DELTA = (long) 5E5;
private static final String CLIENT_TIME_STATISTICS = "CLIENT TIME STATISTICS";
- @CommandLine.Option(names = {"-mode"}, required = true,
- description = "Mode of query runner (singleThread|multiThreads|targetQPS|increasingQPS).")
+ @CommandLine.Option(names = {"-mode"}, required = true, description = "Mode of query runner "
+ + "(singleThread|multiThreads|targetQPS|increasingQPS).")
Review Comment:
just did, no changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org