You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/04/14 00:02:34 UTC

[GitHub] [pinot] npawar opened a new pull request, #8539: Add brokerURL and better handle TimeoutException in QueryRunner

npawar opened a new pull request, #8539:
URL: https://github.com/apache/pinot/pull/8539

   Add support for QueryRunner with brokerURL 
   Don't throw exception if reached configured timeout, instead proceed as normal to print final statictics


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r850688941


##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -416,42 +422,49 @@ public static QuerySummary multiThreadedQueryRunner(PerfBenchmarkDriverConf conf
     long reportStartTime = startTime;
     int numReportIntervals = 0;
     int numTimesExecuted = 0;
-    while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
-      if (executorService.isTerminated()) {
-        throw new IllegalThreadStateException("All threads got exception and already dead.");
-      }
-
-      if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
-        throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
-      }
+    try {
+      while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
+        if (executorService.isTerminated()) {
+          throw new IllegalThreadStateException("All threads got exception and already dead.");
+        }
 
-      for (String query : queries) {
-        while (!queryQueue.offer(query)) {
-          Thread.sleep(1);
+        if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
+          throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
         }
 
-        long currentTime = System.currentTimeMillis();
-        if (currentTime - reportStartTime >= reportIntervalMs) {
-          long timePassed = currentTime - startTime;
-          int numQueriesExecutedInt = numQueriesExecuted.get();
-          LOGGER.info("Time Passed: {}ms, Queries Executed: {}, Exceptions: {}, Average QPS: {}, "
-                  + "Average Broker Time: {}ms, Average Client Time: {}ms.", timePassed, numQueriesExecutedInt,
-              numExceptions.get(), numQueriesExecutedInt / ((double) timePassed / MILLIS_PER_SECOND),
-              totalBrokerTime.get() / (double) numQueriesExecutedInt,
-              totalClientTime.get() / (double) numQueriesExecutedInt);
-          reportStartTime = currentTime;
-          numReportIntervals++;
-
-          if ((numIntervalsToReportAndClearStatistics != 0) && (numReportIntervals
-              == numIntervalsToReportAndClearStatistics)) {
-            numReportIntervals = 0;
-            startTime = currentTime;
-            reportAndClearStatistics(numQueriesExecuted, numExceptions, totalBrokerTime, totalClientTime,
-                statisticsList);
+        for (String query : queries) {
+          if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
+            throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
+          }
+          while (!queryQueue.offer(query)) {
+            Thread.sleep(1);
+          }
+
+          long currentTime = System.currentTimeMillis();
+          if (currentTime - reportStartTime >= reportIntervalMs) {
+            long timePassed = currentTime - startTime;
+            int numQueriesExecutedInt = numQueriesExecuted.get();
+            LOGGER.info("Time Passed: {}ms, Queries Executed: {}, Exceptions: {}, Average QPS: {}, " + "Average "
+                    + "Broker Time: {}ms, Average Client Time: {}ms.", timePassed, numQueriesExecutedInt,
+                numExceptions.get(), numQueriesExecutedInt / ((double) timePassed / MILLIS_PER_SECOND),
+                totalBrokerTime.get() / (double) numQueriesExecutedInt,
+                totalClientTime.get() / (double) numQueriesExecutedInt);
+            reportStartTime = currentTime;
+            numReportIntervals++;
+
+            if ((numIntervalsToReportAndClearStatistics != 0) && (numReportIntervals
+                == numIntervalsToReportAndClearStatistics)) {
+              numReportIntervals = 0;
+              startTime = currentTime;
+              reportAndClearStatistics(numQueriesExecuted, numExceptions, totalBrokerTime, totalClientTime,
+                  statisticsList);
+            }
           }
         }
+        numTimesExecuted++;
       }
-      numTimesExecuted++;
+    } catch (TimeoutException e) {
+      LOGGER.info("Time out reached. {}", e.getMessage());

Review Comment:
   It's not. Event after this change, you'll still get the message `Time out reached. Timeout of 300000 sec reached. Aborting`
   Prior to this change, after this an exception was thrown and no final statistics were printed.
   After this change, we stop gracefully ad print the final stats.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r849984702


##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -299,50 +302,53 @@ public static QuerySummary singleThreadedQueryRunner(PerfBenchmarkDriverConf con
     long reportStartTime = startTime;
     int numReportIntervals = 0;
     int numTimesExecuted = 0;
-    while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
-      for (String query : queries) {
-        if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
-          LOGGER.warn("Timeout of {} sec reached. Aborting", timeout);
-          throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
-        }
+    try {

Review Comment:
   there's no change here other than adding TimeoutException and catching it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r852320632


##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -53,56 +52,58 @@ public class QueryRunner extends AbstractBaseCommand implements Command {
   private static final long NANO_DELTA = (long) 5E5;
   private static final String CLIENT_TIME_STATISTICS = "CLIENT TIME STATISTICS";
 
-  @CommandLine.Option(names = {"-mode"}, required = true,
-      description = "Mode of query runner (singleThread|multiThreads|targetQPS|increasingQPS).")
+  @CommandLine.Option(names = {"-mode"}, required = true, description = "Mode of query runner "
+      + "(singleThread|multiThreads|targetQPS|increasingQPS).")

Review Comment:
   These changes do not seem to improve readability. IIRC we have disabled these kind of auto-reformat in the `Pinot Style`, can you try re-importing the style sheet?



##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -295,15 +297,18 @@ public static QuerySummary singleThreadedQueryRunner(PerfBenchmarkDriverConf con
     List<Statistics> statisticsList = Collections.singletonList(new Statistics(CLIENT_TIME_STATISTICS));
 
     final long startTimeAbsolute = System.currentTimeMillis();
+    boolean timeoutReached = false;
     long startTime = System.currentTimeMillis();
     long reportStartTime = startTime;
     int numReportIntervals = 0;
     int numTimesExecuted = 0;
-    while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
+
+    while (!timeoutReached && (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries)) {
       for (String query : queries) {
         if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
-          LOGGER.warn("Timeout of {} sec reached. Aborting", timeout);
-          throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
+          LOGGER.info("Timeout of {} sec reached. Aborting", timeout);

Review Comment:
   Suggest keeping it as warning. Same for other places



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r852433622


##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -295,15 +297,18 @@ public static QuerySummary singleThreadedQueryRunner(PerfBenchmarkDriverConf con
     List<Statistics> statisticsList = Collections.singletonList(new Statistics(CLIENT_TIME_STATISTICS));
 
     final long startTimeAbsolute = System.currentTimeMillis();
+    boolean timeoutReached = false;
     long startTime = System.currentTimeMillis();
     long reportStartTime = startTime;
     int numReportIntervals = 0;
     int numTimesExecuted = 0;
-    while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
+
+    while (!timeoutReached && (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries)) {
       for (String query : queries) {
         if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
-          LOGGER.warn("Timeout of {} sec reached. Aborting", timeout);
-          throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
+          LOGGER.info("Timeout of {} sec reached. Aborting", timeout);

Review Comment:
   imo this should be info. It is doing exactly what the config is asking it to do, and it ending here isn't an erroneous condition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r853554426


##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -53,56 +52,58 @@ public class QueryRunner extends AbstractBaseCommand implements Command {
   private static final long NANO_DELTA = (long) 5E5;
   private static final String CLIENT_TIME_STATISTICS = "CLIENT TIME STATISTICS";
 
-  @CommandLine.Option(names = {"-mode"}, required = true,
-      description = "Mode of query runner (singleThread|multiThreads|targetQPS|increasingQPS).")
+  @CommandLine.Option(names = {"-mode"}, required = true, description = "Mode of query runner "
+      + "(singleThread|multiThreads|targetQPS|increasingQPS).")

Review Comment:
   ah i see! will keep in mind for next time



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8539:
URL: https://github.com/apache/pinot/pull/8539#issuecomment-1098603629

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#8539](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b1f25ce) into [master](https://codecov.io/gh/apache/pinot/commit/476679da861f20b7c7b1100f818a03eae60a9776?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (476679d) will **increase** coverage by `41.43%`.
   > The diff coverage is `n/a`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #8539       +/-   ##
   =============================================
   + Coverage     29.10%   70.53%   +41.43%     
   - Complexity        0     4290     +4290     
   =============================================
     Files          1669     1681       +12     
     Lines         87585    87949      +364     
     Branches      13274    13317       +43     
   =============================================
   + Hits          25491    62038    +36547     
   + Misses        59770    21601    -38169     
   - Partials       2324     4310     +1986     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `26.87% <ø> (+0.04%)` | :arrow_up: |
   | integration2 | `25.90% <ø> (+0.08%)` | :arrow_up: |
   | unittests1 | `67.02% <ø> (?)` | |
   | unittests2 | `14.06% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | `71.42% <0.00%> (-1.43%)` | :arrow_down: |
   | [...pache/pinot/core/query/utils/idset/EmptyIdSet.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS91dGlscy9pZHNldC9FbXB0eUlkU2V0LmphdmE=) | `25.00% <0.00%> (ø)` | |
   | [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `68.27% <0.00%> (ø)` | |
   | [...anager/realtime/SegmentBuildTimeLeaseExtender.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VnbWVudEJ1aWxkVGltZUxlYXNlRXh0ZW5kZXIuamF2YQ==) | `63.23% <0.00%> (ø)` | |
   | [...in/stream/kafka20/KafkaPartitionLevelConsumer.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL0thZmthUGFydGl0aW9uTGV2ZWxDb25zdW1lci5qYXZh) | `66.66% <0.00%> (ø)` | |
   | [...pache/pinot/ingestion/jobs/SegmentCreationJob.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vam9icy9TZWdtZW50Q3JlYXRpb25Kb2IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [.../pinot/ingestion/jobs/SegmentPreprocessingJob.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vam9icy9TZWdtZW50UHJlcHJvY2Vzc2luZ0pvYi5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...api/resources/ControllerEchoWithAutoDiscovery.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtaW50ZWdyYXRpb24tdGVzdHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L2NvbnRyb2xsZXIvYXBpL3Jlc291cmNlcy9Db250cm9sbGVyRWNob1dpdGhBdXRvRGlzY292ZXJ5LmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...apache/pinot/ingestion/jobs/SegmentTarPushJob.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vam9icy9TZWdtZW50VGFyUHVzaEpvYi5qYXZh) | `70.27% <0.00%> (ø)` | |
   | [...ker/api/resources/BrokerEchoWithAutoDiscovery.java](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtaW50ZWdyYXRpb24tdGVzdHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L2Jyb2tlci9hcGkvcmVzb3VyY2VzL0Jyb2tlckVjaG9XaXRoQXV0b0Rpc2NvdmVyeS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | ... and [1164 more](https://codecov.io/gh/apache/pinot/pull/8539/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [476679d...b1f25ce](https://codecov.io/gh/apache/pinot/pull/8539?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar merged pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
npawar merged PR #8539:
URL: https://github.com/apache/pinot/pull/8539


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] richardstartin commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
richardstartin commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r850312126


##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -416,42 +422,49 @@ public static QuerySummary multiThreadedQueryRunner(PerfBenchmarkDriverConf conf
     long reportStartTime = startTime;
     int numReportIntervals = 0;
     int numTimesExecuted = 0;
-    while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
-      if (executorService.isTerminated()) {
-        throw new IllegalThreadStateException("All threads got exception and already dead.");
-      }
-
-      if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
-        throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
-      }
+    try {
+      while (numTimesToRunQueries == 0 || numTimesExecuted < numTimesToRunQueries) {
+        if (executorService.isTerminated()) {
+          throw new IllegalThreadStateException("All threads got exception and already dead.");
+        }
 
-      for (String query : queries) {
-        while (!queryQueue.offer(query)) {
-          Thread.sleep(1);
+        if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
+          throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
         }
 
-        long currentTime = System.currentTimeMillis();
-        if (currentTime - reportStartTime >= reportIntervalMs) {
-          long timePassed = currentTime - startTime;
-          int numQueriesExecutedInt = numQueriesExecuted.get();
-          LOGGER.info("Time Passed: {}ms, Queries Executed: {}, Exceptions: {}, Average QPS: {}, "
-                  + "Average Broker Time: {}ms, Average Client Time: {}ms.", timePassed, numQueriesExecutedInt,
-              numExceptions.get(), numQueriesExecutedInt / ((double) timePassed / MILLIS_PER_SECOND),
-              totalBrokerTime.get() / (double) numQueriesExecutedInt,
-              totalClientTime.get() / (double) numQueriesExecutedInt);
-          reportStartTime = currentTime;
-          numReportIntervals++;
-
-          if ((numIntervalsToReportAndClearStatistics != 0) && (numReportIntervals
-              == numIntervalsToReportAndClearStatistics)) {
-            numReportIntervals = 0;
-            startTime = currentTime;
-            reportAndClearStatistics(numQueriesExecuted, numExceptions, totalBrokerTime, totalClientTime,
-                statisticsList);
+        for (String query : queries) {
+          if (timeout > 0 && System.currentTimeMillis() - startTimeAbsolute > timeout) {
+            throw new TimeoutException("Timeout of " + timeout + " sec reached. Aborting");
+          }
+          while (!queryQueue.offer(query)) {
+            Thread.sleep(1);
+          }
+
+          long currentTime = System.currentTimeMillis();
+          if (currentTime - reportStartTime >= reportIntervalMs) {
+            long timePassed = currentTime - startTime;
+            int numQueriesExecutedInt = numQueriesExecuted.get();
+            LOGGER.info("Time Passed: {}ms, Queries Executed: {}, Exceptions: {}, Average QPS: {}, " + "Average "
+                    + "Broker Time: {}ms, Average Client Time: {}ms.", timePassed, numQueriesExecutedInt,
+                numExceptions.get(), numQueriesExecutedInt / ((double) timePassed / MILLIS_PER_SECOND),
+                totalBrokerTime.get() / (double) numQueriesExecutedInt,
+                totalClientTime.get() / (double) numQueriesExecutedInt);
+            reportStartTime = currentTime;
+            numReportIntervals++;
+
+            if ((numIntervalsToReportAndClearStatistics != 0) && (numReportIntervals
+                == numIntervalsToReportAndClearStatistics)) {
+              numReportIntervals = 0;
+              startTime = currentTime;
+              reportAndClearStatistics(numQueriesExecuted, numExceptions, totalBrokerTime, totalClientTime,
+                  statisticsList);
+            }
           }
         }
+        numTimesExecuted++;
       }
-      numTimesExecuted++;
+    } catch (TimeoutException e) {
+      LOGGER.info("Time out reached. {}", e.getMessage());

Review Comment:
   timeouts should not be omitted from the reported statistics
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r853540172


##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -53,56 +52,58 @@ public class QueryRunner extends AbstractBaseCommand implements Command {
   private static final long NANO_DELTA = (long) 5E5;
   private static final String CLIENT_TIME_STATISTICS = "CLIENT TIME STATISTICS";
 
-  @CommandLine.Option(names = {"-mode"}, required = true,
-      description = "Mode of query runner (singleThread|multiThreads|targetQPS|increasingQPS).")
+  @CommandLine.Option(names = {"-mode"}, required = true, description = "Mode of query runner "
+      + "(singleThread|multiThreads|targetQPS|increasingQPS).")

Review Comment:
   The new one will honor the new lines and not try to merge separate lines, so if you revert and reformat, it won't apply the same change here. Not important though :-P



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar commented on a diff in pull request #8539: Add brokerURL and better handle TimeoutException in QueryRunner

Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8539:
URL: https://github.com/apache/pinot/pull/8539#discussion_r852432915


##########
pinot-tools/src/main/java/org/apache/pinot/tools/perf/QueryRunner.java:
##########
@@ -53,56 +52,58 @@ public class QueryRunner extends AbstractBaseCommand implements Command {
   private static final long NANO_DELTA = (long) 5E5;
   private static final String CLIENT_TIME_STATISTICS = "CLIENT TIME STATISTICS";
 
-  @CommandLine.Option(names = {"-mode"}, required = true,
-      description = "Mode of query runner (singleThread|multiThreads|targetQPS|increasingQPS).")
+  @CommandLine.Option(names = {"-mode"}, required = true, description = "Mode of query runner "
+      + "(singleThread|multiThreads|targetQPS|increasingQPS).")

Review Comment:
   just did, no changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org