[GitHub] [druid] jasonk000 commented on a diff in pull request #12102: HttpPostEmitter back off send() busy-loop

[GitBox <gi...@apache.org>]

jasonk000 commented on code in PR #12102:
URL: https://github.com/apache/druid/pull/12102#discussion_r971009152


##########
core/src/main/java/org/apache/druid/java/util/emitter/core/HttpPostEmitter.java:
##########
@@ -698,13 +698,42 @@ public boolean apply(Throwable e)
       catch (Exception e) {
         if (e == timeoutLessThanMinimumException) {
           log.debug(e, "Failed to send events to url[%s] with timeout less than minimum", config.getRecipientBaseUrl());
+          sendBackoffDelay();
         } else {
           log.error(e, "Failed to send events to url[%s]", config.getRecipientBaseUrl());
         }
         return false;
       }
     }
 
+    /**
+     * Back-off the sender for a period of time.
+     *
+     * Call only from the send() loop thread when it sees a timeout.
+     */
+    void sendBackoffDelay()
+    {
+      // The backoff delay is quite simple in favour of something more complicated.
+
+      // This is called when the send() call hits a timeoutLessThanMinimumException
+      // condition. Ideally, what we should do is delay until either the timeout increases
+      // to above the minHttpTimeout, and so the send() will be issued, or, until a Batch
+      // wants to flush based on the configured timeout.
+
+      // Making this technically correct, across all possible Batches, including large events,
+      // etc is complicated in this codebase, since batch flush timing is not tracked here,
+      // so rather than introduce new bugs, let's keep it simple.
+
+      final long backoffCheckDelayMillis = config.getMinHttpTimeoutMillis() / 5;

Review Comment:
   Hi @FrankChen021 were you able to review / confirm suitability of the approach here?
   
   CC @abhishekagarwal87 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org