You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "rohangarg (via GitHub)" <gi...@apache.org> on 2023/05/03 12:17:09 UTC

[GitHub] [druid] rohangarg opened a new pull request, #14204: Add more logging for query timeouts and errors during broker parallel merge

rohangarg opened a new pull request, #14204:
URL: https://github.com/apache/druid/pull/14204

   This change adds a couple of changes : 
   1. Prefers to throw any cancellations or execution errors in the parallel merge over the query timeout exception
   2. Adds some state logging of the parallel merge incase a timeout occurs - this is helpful in debugging which state the merge was stuck in
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add more logging for query timeouts and errors during broker parallel merge (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14204:
URL: https://github.com/apache/druid/pull/14204#issuecomment-1989680228

   This pull request/issue has been closed due to lack of activity. If you think that
   is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #14204: Add more logging for query timeouts and errors during broker parallel merge

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #14204:
URL: https://github.com/apache/druid/pull/14204#discussion_r1230710308


##########
processing/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java:
##########
@@ -1310,9 +1327,22 @@ public long getSlowestPartitionInitializedTime()
 
     private final int inputSequences;
 
+    public enum State
+    {
+      SCHEDULED,
+      STARTED,
+      PARTITON_MERGE_SCHEDULED,
+      FINAL_MERGE_SCHEDULED
+    }
+
+    private State state;
+    private long stateStartTime;
+
     MergeCombineMetricsAccumulator(int inputSequences)
     {
       this.inputSequences = inputSequences;
+      this.partitionMetrics = Collections.emptyList();
+      this.mergeMetrics = new MergeCombineActionMetricsAccumulator();

Review Comment:
   #14426 also initializes these, though externally, so we should either remove those external calls or these constructor initializations (i think its a bit cleaner to be handled in the constructor like is done in this PR, but don't care too much either way)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add more logging for query timeouts and errors during broker parallel merge (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14204:
URL: https://github.com/apache/druid/pull/14204#issuecomment-1937927782

   This pull request has been marked as stale due to 60 days of inactivity.
   It will be closed in 4 weeks if no further activity occurs. If you think
   that's incorrect or this pull request should instead be reviewed, please simply
   write any comment. Even if closed, you can still revive the PR at any time or
   discuss it on the dev@druid.apache.org list.
   Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add more logging for query timeouts and errors during broker parallel merge (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14204: Add more logging for query timeouts and errors during broker parallel merge
URL: https://github.com/apache/druid/pull/14204


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #14204: Add more logging for query timeouts and errors during broker parallel merge

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #14204:
URL: https://github.com/apache/druid/pull/14204#discussion_r1190646105


##########
processing/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java:
##########
@@ -1310,9 +1327,22 @@ public long getSlowestPartitionInitializedTime()
 
     private final int inputSequences;
 
+    public enum State
+    {
+      SCHEDULED,
+      STARTED,
+      PARTITON_MERGE_SCHEDULED,
+      FINAL_MERGE_SCHEDULED
+    }

Review Comment:
   could you explain the states a bit better? Like, the tasks from both layers of the merge are all started at basically the same time, and keep scheduling additional actions if there is more stuff to process until their inputs are completely drained. So I don't really understand how the state transitions are supposed to work and what they are supposed to tell us. It seems like unless it gets blocked completely at SCHEDULED, chances are it will burn through the other two scheduled states like immediately (since its just adding tasks to a queue, not blocked on them actually running afaict). Im not really sure there are meaningful states between haven't started, running, and done, and done isn't very interesting. More interesting seems like the information about the number of tasks each partition has run and/or number of rows they have processed, vs amount of stuff processed by the 2nd layer "final merge" tasks (like is it being held up by a slow partition, etc), but I'm not really sur
 e how to easily wrap that stuff up in state.



##########
processing/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java:
##########
@@ -191,6 +194,11 @@ public boolean hasNext()
               {
                 final long thisTimeoutNanos = timeoutAtNanos - System.nanoTime();
                 if (hasTimeout && thisTimeoutNanos < 0) {
+                  LOG.info("Query timed out while merging results from data nodes. "
+                           + "Current merge state is [%s] and time spent in current state is [%,d]ms.",
+                           mergeCombineMetricsAccumulator.getState(),
+                           mergeCombineMetricsAccumulator.timeSpentInCurrentStateNanos()
+                  );

Review Comment:
   same though on info logs



##########
processing/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java:
##########
@@ -205,14 +213,20 @@ public boolean hasNext()
                       currentBatch = queue.take();
                     }
                   }
-                  if (currentBatch == null) {
-                    throw new QueryTimeoutException();
-                  }
 
-                  if (cancellationGizmo.isCancelled()) {
+                  if (cancellationGizmo.isCancelled()) { // preffering to throw any cancellation exception, if any
                     throw cancellationGizmo.getRuntimeException();
                   }
 
+                  if (currentBatch == null) {
+                    LOG.info("Query timed out while merging results from data nodes. "
+                             + "Current merge state is [%s] and time spent in current state is [%,d]ms.",
+                             mergeCombineMetricsAccumulator.getState(),
+                             mergeCombineMetricsAccumulator.timeSpentInCurrentStateNanos()
+                    );

Review Comment:
   info doesn't really seem right for this, should this information just decorate the timeout exception instead since that is also going to be logged?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #14204: Add more logging for query timeouts and errors during broker parallel merge

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #14204:
URL: https://github.com/apache/druid/pull/14204#discussion_r1190649229


##########
processing/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java:
##########
@@ -191,6 +194,11 @@ public boolean hasNext()
               {
                 final long thisTimeoutNanos = timeoutAtNanos - System.nanoTime();
                 if (hasTimeout && thisTimeoutNanos < 0) {
+                  LOG.info("Query timed out while merging results from data nodes. "
+                           + "Current merge state is [%s] and time spent in current state is [%,d]ms.",
+                           mergeCombineMetricsAccumulator.getState(),
+                           mergeCombineMetricsAccumulator.timeSpentInCurrentStateNanos()
+                  );

Review Comment:
   same thought on info logs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org