You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/07/14 23:57:49 UTC
[GitHub] [kafka] mjsax opened a new pull request #11055: HOTFIX: Init stream-stream left/outer join emit interval correctly
mjsax opened a new pull request #11055:
URL: https://github.com/apache/kafka/pull/11055
Follow up to KAFKA-10847 (https://github.com/apache/kafka/pull/10917).
The above fix intended to reduce the emit frequency to save the creation cost of RocksDB iterators. However, we incorrectly initialized the "timer" with timestamp zero, and thus, the timer was always in the past and we did try to emit left/outer join result too often and got a throughput of only 500 record/sec.
This PR fixes the initialization of the emit interval timer. After re-running the benchmark, we determined that a default emit interval of 1000ms provides the best performance of 12K rec/sec, so this PR also changes the emit frequency to 1000ms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #11055: HOTFIX: Init stream-stream left/outer join emit interval correctly
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #11055:
URL: https://github.com/apache/kafka/pull/11055#issuecomment-880286834
@kkonstantine I think we should try to get this into 3.0 release. Thoughts?
Call for review @guozhangwang @spena
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax merged pull request #11055: HOTFIX: Init stream-stream left/outer join emit interval correctly
Posted by GitBox <gi...@apache.org>.
mjsax merged pull request #11055:
URL: https://github.com/apache/kafka/pull/11055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #11055: HOTFIX: Init stream-stream left/outer join emit interval correctly
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #11055:
URL: https://github.com/apache/kafka/pull/11055#discussion_r670030416
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamKStreamJoin.java
##########
@@ -103,7 +103,6 @@ public void init(final org.apache.kafka.streams.processor.ProcessorContext conte
true
)) {
outerJoinWindowStore = outerJoinWindowName.map(context::getStateStore);
- sharedTimeTracker.nextTimeToEmit = context.currentSystemTimeMs();
Review comment:
In `init()` the cached system time is still zero, so we need to delay the initialization to a later point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #11055: HOTFIX: Init stream-stream left/outer join emit interval correctly
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #11055:
URL: https://github.com/apache/kafka/pull/11055#issuecomment-881729479
Merged to `trunk` and cherry-picked to `3.0` branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org