You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/01/12 20:09:44 UTC
[GitHub] [kafka] ableegoldman opened a new pull request #9875: MINOR: tighten condition for logging StreamThread summary to avoid spam under low traffic
ableegoldman opened a new pull request #9875:
URL: https://github.com/apache/kafka/pull/9875
Noticed these messages flooding the logs on several integration tests. Since it doesn't really provide any useful information, we may as well skip the logging when there are no new records to be processed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wcarlson5 commented on a change in pull request #9875: MINOR: tighten condition for logging StreamThread summary to avoid spam under low traffic
Posted by GitBox <gi...@apache.org>.
wcarlson5 commented on a change in pull request #9875:
URL: https://github.com/apache/kafka/pull/9875#discussion_r556094266
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java
##########
@@ -824,7 +826,9 @@ private long pollPhase() {
final long pollLatency = advanceNowAndComputeLatency();
final int numRecords = records.count();
- log.info("Main Consumer poll completed in {} ms and fetched {} records", pollLatency, numRecords);
+ if (numRecords > 0) {
Review comment:
The above probably doesn't need to be applied here as well, i don't think
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java
##########
@@ -749,9 +749,11 @@ void runOnce() {
// multiple iterations with reasonably large max.num.records and hence is less vulnerable to outliers
taskManager.recordTaskProcessRatio(totalProcessLatency, now);
- log.info("Processed {} total records, ran {} punctuators, and committed {} total tasks " +
- "for active tasks {} and standby tasks {}",
- totalProcessed, totalPunctuated, totalCommitted, taskManager.activeTaskIds(), taskManager.standbyTaskIds());
+ // Don't log summary if no new records were processed to avoid spamming logs for low-traffic topics
+ if (totalProcessed > 0 || totalPunctuated > 0 || totalCommitted > 0) {
+ log.info("Processed {} total records, ran {} punctuators, and committed {} total tasks",
Review comment:
Maybe we do this else where but can we log at DEBUG if there is nothing done and there is tasks assigned to the thread? It might be good to be able to confirm that its not getting stuck here or its actually not polling anything.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9875: MINOR: tighten condition for logging StreamThread summary to avoid spam under low traffic
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9875:
URL: https://github.com/apache/kafka/pull/9875#discussion_r556146003
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java
##########
@@ -749,9 +749,11 @@ void runOnce() {
// multiple iterations with reasonably large max.num.records and hence is less vulnerable to outliers
taskManager.recordTaskProcessRatio(totalProcessLatency, now);
- log.info("Processed {} total records, ran {} punctuators, and committed {} total tasks " +
- "for active tasks {} and standby tasks {}",
- totalProcessed, totalPunctuated, totalCommitted, taskManager.activeTaskIds(), taskManager.standbyTaskIds());
+ // Don't log summary if no new records were processed to avoid spamming logs for low-traffic topics
+ if (totalProcessed > 0 || totalPunctuated > 0 || totalCommitted > 0) {
+ log.info("Processed {} total records, ran {} punctuators, and committed {} total tasks",
Review comment:
I have a similar question here, other than that LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #9875: MINOR: tighten condition for logging StreamThread summary to avoid spam under low traffic
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #9875:
URL: https://github.com/apache/kafka/pull/9875#issuecomment-759782754
Merged to trunk and cherrypicked to 2.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9875: MINOR: tighten condition for logging StreamThread summary to avoid spam under low traffic
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9875:
URL: https://github.com/apache/kafka/pull/9875#discussion_r556910001
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java
##########
@@ -749,9 +749,11 @@ void runOnce() {
// multiple iterations with reasonably large max.num.records and hence is less vulnerable to outliers
taskManager.recordTaskProcessRatio(totalProcessLatency, now);
- log.info("Processed {} total records, ran {} punctuators, and committed {} total tasks " +
- "for active tasks {} and standby tasks {}",
- totalProcessed, totalPunctuated, totalCommitted, taskManager.activeTaskIds(), taskManager.standbyTaskIds());
+ // Don't log summary if no new records were processed to avoid spamming logs for low-traffic topics
+ if (totalProcessed > 0 || totalPunctuated > 0 || totalCommitted > 0) {
+ log.info("Processed {} total records, ran {} punctuators, and committed {} total tasks",
Review comment:
Good thought, but don't worry, we already do. John added some pretty thorough logging at the DEBUG level after an escalation a while back
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman merged pull request #9875: MINOR: tighten condition for logging StreamThread summary to avoid spam under low traffic
Posted by GitBox <gi...@apache.org>.
ableegoldman merged pull request #9875:
URL: https://github.com/apache/kafka/pull/9875
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #9875: MINOR: tighten condition for logging StreamThread summary to avoid spam under low traffic
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #9875:
URL: https://github.com/apache/kafka/pull/9875#issuecomment-758913410
call for a quick review from any of @cadonna @lct45 @wcarlson5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org