You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/07/01 03:01:02 UTC
[GitHub] [kafka] ableegoldman opened a new pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
ableegoldman opened a new pull request #8963:
URL: https://github.com/apache/kafka/pull/8963
The current failures we're seeing with this test are due to faulty assumptions that it makes and not any real bug in eos-beta (at least, from what I've seen so far).
The test relies on tightly controlling the commits, which it does by setting the commit interval to MAX_VALUE and manually requesting commits on the context. In two phases, the test assumes that any pending data will be committed after a rebalance. But we actually take care to avoid unnecessary commits -- with eos-alpha, we only commit tasks that are revoked while in eos-beta we must commit all tasks if any are revoked, but only if the revoked tasks themselves need a commit.
The failure we see occurs when we try to verify the committed data after a second client is started and the group rebalances. The already-running client has to give up two tasks to the newly started client, but those tasks may not need to be committed in which case none of the tasks would be. So we still have an open transaction on the partitions where we try to read committed data.
We can use a punctuator to force a commit on the running client
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman edited a comment on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman edited a comment on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-652773774
Looks like it only failed one time across all three builds in the last run. But I pushed some fixes for the remaining flakiness:
1. Use the IntegrationTestUtils `#getStore` to retry on InvalidStateStoreException
2. Relax the assignment listener check: the way we currently make sure to reach a stable assignment is to save the current number of stable assignments before a rebalance-triggering event (eg 2nd client is stopped) and then wait for the number of stable assignments to increase by exactly one. In some cases the test failed because we actually got to more than the expected number of stable assignments. I'm not exactly sure why, but I don't think it's necessarily a problem and we can probably just relax the condition to make sure we had at least one more stable assignment
Can you kick off the tests again @guozhangwang ? I'm interested to see if the fix for 2. is enough or if whatever is causing the extra rebalance will surface as a test failure in another way
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-652709044
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653235610
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-652773774
Looks like it only failed one time across all three builds in the last run. But I pushed some fixes for the remaining flakiness:
1. Use the IntegrationTestUtils `#getStore` to retry on InvalidStateStoreException
2. Relax the assignment listener check: the way we currently make sure to reach a stable assignment is to save the current number of stable assignments before a rebalance-triggering event (eg 2nd client is stopped) and then wait for the number of stable assignments to increase by exactly one. In some cases the test failed because we actually got to more than the expected number of stable assignments. I'm not exactly sure why, but I don't think it's necessarily a problem and we can probably just relax the condition to make sure we had at least one more stable assignment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#discussion_r449281005
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -147,6 +152,25 @@
private final AtomicInteger commitCounterClient2 = new AtomicInteger(-1);
private final AtomicInteger commitRequested = new AtomicInteger(0);
+ private final AtomicBoolean requestCommit = new AtomicBoolean(false);
+ private static class CommitPunctuator implements Punctuator {
+ final ProcessorContext context;
+ final AtomicBoolean requestCommit;
+
+ public CommitPunctuator(final ProcessorContext context, final AtomicBoolean requestCommit) {
+ this.context = context;
+ this.requestCommit = requestCommit;
+ }
+
+ @Override
+ public void punctuate(final long timestamp) {
+ if (requestCommit.get()) {
+ context.commit();
+ requestCommit.set(false);
+ }
Review comment:
sounds good
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-654472201
Merged to 2.6 as well and re-enabled the integration test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a change in pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on a change in pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#discussion_r449798211
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -837,6 +866,11 @@ public void init(final ProcessorContext context) {
crash = errorInjectedClient2;
sharedCommit = commitCounterClient2;
}
+ punctuator = context.schedule(
Review comment:
In this test we would have multiple punctuator indeed but they would be executed by a single thread sequentially so that's fine.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/utils/IntegrationTestUtils.java
##########
@@ -1273,7 +1273,7 @@ public void prepareForRebalance() {
*/
public void waitForNextStableAssignment(final long maxWaitMs) throws InterruptedException {
waitForCondition(
- () -> nextExpectedNumStableAssignments == numStableAssignments(),
+ () -> numStableAssignments() >= nextExpectedNumStableAssignments,
Review comment:
Makes sense to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#discussion_r449220447
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -147,6 +152,25 @@
private final AtomicInteger commitCounterClient2 = new AtomicInteger(-1);
private final AtomicInteger commitRequested = new AtomicInteger(0);
+ private final AtomicBoolean requestCommit = new AtomicBoolean(false);
+ private static class CommitPunctuator implements Punctuator {
+ final ProcessorContext context;
+ final AtomicBoolean requestCommit;
+
+ public CommitPunctuator(final ProcessorContext context, final AtomicBoolean requestCommit) {
+ this.context = context;
+ this.requestCommit = requestCommit;
+ }
+
+ @Override
+ public void punctuate(final long timestamp) {
+ if (requestCommit.get()) {
+ context.commit();
+ requestCommit.set(false);
+ }
Review comment:
This is only meant to be used when there's a single thread (and single instance running). I just wanted to put in a quick fix for now--I'll leave a comment to warn our future selves that we'll have to tighten this up if we want to extend the test or use it elsewhere with multiple threads/instances
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653235811
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman edited a comment on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman edited a comment on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653176567
> Seems still got 3 failures, while 1/3 green
@abbccdda Did the EosBetaUpgradeIntegrationTest fail 3 times? Did you get the link or stacktrace before kicking off tests again?
edit: I assume that was the failure you just linked to on the flaky tests ticket? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#discussion_r449282334
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -837,6 +866,11 @@ public void init(final ProcessorContext context) {
crash = errorInjectedClient2;
sharedCommit = commitCounterClient2;
}
+ punctuator = context.schedule(
Review comment:
Ok, I was thinking more along the lines that we have just one requestCommit instance while we would have multiple punctuator instances all using it. It looked like a bug, but it seems like it'll be fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653236124
When we merge this, we should cherry-pick it to 2.6 and remove the ignore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] abbccdda commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
abbccdda commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-652159442
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#discussion_r449219621
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -837,6 +866,11 @@ public void init(final ProcessorContext context) {
crash = errorInjectedClient2;
sharedCommit = commitCounterClient2;
}
+ punctuator = context.schedule(
Review comment:
Sort of; with eos-beta when you need to commit one task, you need to commit all of them, so requesting a commit at all should be sufficient. It's kind of a subtle point, but I think it's actually preferable to just request a commit on one task so this test can also verify that we commit all tasks together
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653036631
Test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653176567
> Seems still got 3 failures, while 1/3 green
@abbccdda Did the EosBetaUpgradeIntegrationTest fail 3 times? Did you get the link or stacktrace before kicking off tests again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang merged pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
guozhangwang merged pull request #8963:
URL: https://github.com/apache/kafka/pull/8963
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#discussion_r449223048
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -837,6 +866,11 @@ public void init(final ProcessorContext context) {
crash = errorInjectedClient2;
sharedCommit = commitCounterClient2;
}
+ punctuator = context.schedule(
+ Duration.ofSeconds(5),
Review comment:
I suppose it would. Does 100ms seem reasonable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#discussion_r449041784
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -837,6 +866,11 @@ public void init(final ProcessorContext context) {
crash = errorInjectedClient2;
sharedCommit = commitCounterClient2;
}
+ punctuator = context.schedule(
+ Duration.ofSeconds(5),
Review comment:
Would it speed up the test to choose a smaller number here?
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -837,6 +866,11 @@ public void init(final ProcessorContext context) {
crash = errorInjectedClient2;
sharedCommit = commitCounterClient2;
}
+ punctuator = context.schedule(
Review comment:
There's going to be a separate punctuator per task, right? Does the test account for this?
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -147,6 +152,25 @@
private final AtomicInteger commitCounterClient2 = new AtomicInteger(-1);
private final AtomicInteger commitRequested = new AtomicInteger(0);
+ private final AtomicBoolean requestCommit = new AtomicBoolean(false);
+ private static class CommitPunctuator implements Punctuator {
+ final ProcessorContext context;
+ final AtomicBoolean requestCommit;
+
+ public CommitPunctuator(final ProcessorContext context, final AtomicBoolean requestCommit) {
+ this.context = context;
+ this.requestCommit = requestCommit;
+ }
+
+ @Override
+ public void punctuate(final long timestamp) {
+ if (requestCommit.get()) {
+ context.commit();
+ requestCommit.set(false);
+ }
Review comment:
There should never be multiple requests, right? If there were, a second request might arrive between 168 and 169, violating the desired property. In that case, we should grab a lock instead. As long as there's only one requesting thread, and it always waits for the commit right after requesting, then we should be good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-654347195
@guozhangwang I don't _think_ we need [8962](https://github.com/apache/kafka/pull/8962) to be merged first
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-652494651
EosBetaUpgradeIntegrationTest still failed a few times on the PR builds, but it seems to be actual flakiness and not a new bug in eos-beta. I'll try and tighten up the test a bit more
`org.apache.kafka.streams.errors.InvalidStateStoreException: Cannot get state store store because the stream thread is PARTITIONS_ASSIGNED, not RUNNING`
and
`java.lang.AssertionError: Condition not met within timeout 60000. Client did not reach 10 stable assignments on time, numStableAssignments was 11`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653241923
It would be good to run this in a loop locally 100 times (or something like that) to check that it's truly stable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-652774323
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-652774552
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman edited a comment on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
ableegoldman edited a comment on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653176567
> Seems still got 3 failures, while 1/3 green
@abbccdda Did the EosBetaUpgradeIntegrationTest fail 3 times? Did you get the link or stacktrace before kicking off tests again?
edit: Nevermind, I assume you meant other flaky tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] abbccdda commented on pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
abbccdda commented on pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#issuecomment-653158158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #8963: KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #8963:
URL: https://github.com/apache/kafka/pull/8963#discussion_r449280874
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/EosBetaUpgradeIntegrationTest.java
##########
@@ -837,6 +866,11 @@ public void init(final ProcessorContext context) {
crash = errorInjectedClient2;
sharedCommit = commitCounterClient2;
}
+ punctuator = context.schedule(
+ Duration.ofSeconds(5),
Review comment:
Yeah, that's what I was thinking
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org