You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/07/06 22:07:50 UTC
[GitHub] [druid] harinirajendran opened a new pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
harinirajendran opened a new pull request #11415:
URL: https://github.com/apache/druid/pull/11415
While I was trying to gather more info for https://github.com/apache/druid/issues/11414, I realized we do not have much visibility into the notices processing time inside kafka stream supervisor. Hence adding more debug logs to increase that visibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #11415:
URL: https://github.com/apache/druid/pull/11415#issuecomment-876802788
Sure, I just merged :slightly_smiling_face:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] harinirajendran commented on pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
harinirajendran commented on pull request #11415:
URL: https://github.com/apache/druid/pull/11415#issuecomment-876777657
@jihoonson @xvrl The build failure doesn't seem to be related to my changes. what do I do now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] harinirajendran commented on pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
harinirajendran commented on pull request #11415:
URL: https://github.com/apache/druid/pull/11415#issuecomment-875816910
> Looks good but please address the concern about the two logs being hard to distinguish.
>
> BTW, it seems like a good idea to emit the queue size as a metric as well. We could do it in a follow-up PR.
@xvrl suggested the same as well :) I am working on a PR right now to add it. will raise it once its ready!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] harinirajendran edited a comment on pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
harinirajendran edited a comment on pull request #11415:
URL: https://github.com/apache/druid/pull/11415#issuecomment-875816910
> Looks good but please address the concern about the two logs being hard to distinguish.
>
> BTW, it seems like a good idea to emit the queue size as a metric as well. We could do it in a follow-up PR.
@xvrl suggested the same as well :) I am working on a PR right now to add it. will raise it once its ready!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
jihoonson merged pull request #11415:
URL: https://github.com/apache/druid/pull/11415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] harinirajendran commented on a change in pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
harinirajendran commented on a change in pull request #11415:
URL: https://github.com/apache/druid/pull/11415#discussion_r665611957
##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
##########
@@ -902,7 +902,9 @@ public void tryInit()
}
try {
+ log.debug("Handling notice [%s] from notices queue, current notices queue size [%d]", notice.getClass().getName(), getNoticesQueueSize());
notice.handle();
+ log.debug("Handled notice [%s] from notices queue, current notices queue size [%d]", notice.getClass().getName(), getNoticesQueueSize());
}
Review comment:
The intent for adding the 2 log statements was to get the time taken to process the notices just by looking at the 2 log lines. Now just logged the time difference instead and removed 1 of the log lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #11415:
URL: https://github.com/apache/druid/pull/11415#discussion_r665580840
##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
##########
@@ -902,7 +902,9 @@ public void tryInit()
}
try {
+ log.debug("Handling notice [%s] from notices queue, current notices queue size [%d]", notice.getClass().getName(), getNoticesQueueSize());
notice.handle();
+ log.debug("Handled notice [%s] from notices queue, current notices queue size [%d]", notice.getClass().getName(), getNoticesQueueSize());
}
Review comment:
I agree with @xvrl. It seems enough to me to keep only one of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] harinirajendran commented on pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
harinirajendran commented on pull request #11415:
URL: https://github.com/apache/druid/pull/11415#issuecomment-875836851
@jihoonson @xvrl Just raised https://github.com/apache/druid/pull/11417/files to add the new metric for notices queue size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #11415:
URL: https://github.com/apache/druid/pull/11415#issuecomment-876788908
Those failed tests seem flaky. I restarted them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] harinirajendran commented on pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
harinirajendran commented on pull request #11415:
URL: https://github.com/apache/druid/pull/11415#issuecomment-876793063
> Those failed tests seem flaky. I restarted them.
Thank u @jihoonson ! looks like they have passed now. Can we merge this PR at this point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #11415: Adding more debug logs to increase visibility into StreamSupervisor notices queue size and processing time.
Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #11415:
URL: https://github.com/apache/druid/pull/11415#discussion_r665551190
##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
##########
@@ -902,7 +902,9 @@ public void tryInit()
}
try {
+ log.debug("Handling notice [%s] from notices queue, current notices queue size [%d]", notice.getClass().getName(), getNoticesQueueSize());
notice.handle();
+ log.debug("Handled notice [%s] from notices queue, current notices queue size [%d]", notice.getClass().getName(), getNoticesQueueSize());
}
Review comment:
those two debug messages are very hard to distinguish. I think we should make it more obvious to avoid confusion when debugging, or only keep one of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org