You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/03/03 07:05:28 UTC
[GitHub] [kafka] lkokhreidze opened a new pull request #11837: KAFKA-6718 / Add client.tag. prefix constant in StreamsConfig
lkokhreidze opened a new pull request #11837:
URL: https://github.com/apache/kafka/pull/11837
This PR is part of [KIP-708](https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+awareness+for+Kafka+Streams) and adds rack aware standby task assignment logic.
Rack aware standby task assignment won't be functional until all parts of this KIP gets merged.
Splitting PRs into three smaller PRs to make the review process easier to follow. Overall plan is the following:
⏭️ Rack aware standby task assignment logic https://github.com/apache/kafka/pull/10851
⏭️ Protocol change, add clientTags to SubscriptionInfoData https://github.com/apache/kafka/pull/10802
👉 Add required configurations to StreamsConfig (public API change, at this point we should have full functionality)
This PR implements first point of the above mentioned plan.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1067690817
Thanks for the review @showuon
I've addressed your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826787871
##########
File path: clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java
##########
@@ -1121,6 +1121,27 @@ public String toString() {
}
}
+ public static class ListSize implements Validator {
+ final int maxSize;
+
+ private ListSize(final int maxSize) {
+ this.maxSize = maxSize;
+ }
+
+ public static ListSize atMostOfSize(final int maxSize) {
+ return new ListSize(maxSize);
+ }
+
+ @Override
+ public void ensureValid(final String name, final Object value) {
+ @SuppressWarnings("unchecked")
+ List<String> values = (List<String>) value;
+ if (values.size() > maxSize) {
+ throw new ConfigException(name, value, "exceeds maximum list size of [" + maxSize + "].");
Review comment:
I'm not quite sure if the `value` will output the list strings. Should we use `values` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827825740
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
+1 for 100 tasks. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826914313
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
Thanks for the comment. I will add the integration test to test all three of those configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827960149
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
Thanks @cadonna
If no objections, I will do that now. The fact that it times out in 1 minute doesn't give much confidence that it won't be flaky.
I will follow up the system test.
fyi @showuon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069108584
Thanks @cadonna ,
I assume it's okay to do it in a follow-up PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825755024
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
Do not have any science behind those numbers, but thought it's better to have some restrictions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069358982
Merged!
@lkokhreidze Congrats to this milestone! Nice work! 🥳
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827305741
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
Was able to push up to a two hundred stateful tasks and even that was already slow. After 500, test was getting flaky on my computer and sometimes it was just failing due to timeouts.
Feels like these kind of test is too heavy for a local integration tests. Any other way we could do it? Or any pointers to any of the existing integration tests that do the similar thing?
Anyway, I've pushed the changes. I also found one bug in the `ClientTagAwareStandbyTaskAssignor` which is now fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825779088
##########
File path: clients/src/test/java/org/apache/kafka/common/config/ConfigDefTest.java
##########
@@ -722,4 +723,24 @@ public void testNiceTimeUnits() {
assertEquals(" (365 days)", ConfigDef.niceTimeUnits(Duration.ofDays(365).toMillis()));
}
+ @Test
+ public void testThrowsExceptionWhenListSizeExceedsLimit() {
+ assertThrows(ConfigException.class, () -> new ConfigDef().define("lst",
+ Type.LIST,
+ asList("a", "b"),
+ ListSize.max(1),
+ Importance.HIGH,
+ "lst doc"));
+ }
+
+ @Test
+ public void testNoExceptionIsThrownWhenListSizeIsWithinTheLimit() {
+ new ConfigDef().define("lst",
+ Type.LIST,
+ asList("a", "b"),
+ ListSize.max(2),
+ Importance.HIGH,
+ "lst doc");
+ }
+
Review comment:
Could you add a unit test, so that there is one with a list that is longer than the maximum, one with a list that is strictly shorter than maximum, and one with list that has maximum length?
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.ThreadMetadata;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Transformer;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestCondition;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static java.util.Arrays.asList;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+
+@Category({IntegrationTest.class})
+public class RackAwarenessIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ private static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ @Rule
+ public TestName testName = new TestName();
+
+ private static final String INPUT_TOPIC = "input-topic";
+ private static final String TAG_ZONE = "zone";
+ private static final String TAG_CLUSTER = "cluster";
+
+ private List<KafkaStreamsWithConfiguration> kafkaStreamsInstances;
+ private Properties baseConfiguration;
+
+ @BeforeClass
+ public static void createTopics() throws Exception {
+ CLUSTER.start();
+ CLUSTER.createTopic(INPUT_TOPIC, 6, 1);
+ }
+
+ @Before
+ public void setup() {
+ kafkaStreamsInstances = new ArrayList<>();
+ baseConfiguration = new Properties();
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+ final String applicationId = "app-" + safeTestName;
+ baseConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ baseConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ baseConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ baseConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ baseConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ }
+
+ @After
+ public void cleanup() throws IOException {
+ for (final KafkaStreamsWithConfiguration kafkaStreamsWithConfiguration : kafkaStreamsInstances) {
+ kafkaStreamsWithConfiguration.kafkaStreams.close(Duration.ofMillis(IntegrationTestUtils.DEFAULT_TIMEOUT));
+ IntegrationTestUtils.purgeLocalStreamsState(kafkaStreamsWithConfiguration.configuration);
+ }
+ kafkaStreamsInstances.clear();
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasBasedOnClientTags() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 1;
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilRackAwareTaskDistributionIsReached(TAG_ZONE);
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasOverMultipleClientTags() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 2;
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1c", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-2"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-2"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1c", "k8s-cluster-2"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-3"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-3"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1c", "k8s-cluster-3"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
Review comment:
I think, here it makes sense to wait until all Streams clients are `RUNNING` so that we know that the rebalance is done.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.ThreadMetadata;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Transformer;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestCondition;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static java.util.Arrays.asList;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+
+@Category({IntegrationTest.class})
+public class RackAwarenessIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ private static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ @Rule
+ public TestName testName = new TestName();
+
+ private static final String INPUT_TOPIC = "input-topic";
+ private static final String TAG_ZONE = "zone";
+ private static final String TAG_CLUSTER = "cluster";
+
+ private List<KafkaStreamsWithConfiguration> kafkaStreamsInstances;
+ private Properties baseConfiguration;
+
+ @BeforeClass
+ public static void createTopics() throws Exception {
+ CLUSTER.start();
+ CLUSTER.createTopic(INPUT_TOPIC, 6, 1);
+ }
+
+ @Before
+ public void setup() {
+ kafkaStreamsInstances = new ArrayList<>();
+ baseConfiguration = new Properties();
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+ final String applicationId = "app-" + safeTestName;
+ baseConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ baseConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ baseConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ baseConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ baseConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ }
+
+ @After
+ public void cleanup() throws IOException {
+ for (final KafkaStreamsWithConfiguration kafkaStreamsWithConfiguration : kafkaStreamsInstances) {
+ kafkaStreamsWithConfiguration.kafkaStreams.close(Duration.ofMillis(IntegrationTestUtils.DEFAULT_TIMEOUT));
+ IntegrationTestUtils.purgeLocalStreamsState(kafkaStreamsWithConfiguration.configuration);
+ }
+ kafkaStreamsInstances.clear();
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasBasedOnClientTags() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 1;
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
Review comment:
See my comment below about waiting for `RUNNING`.
##########
File path: clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java
##########
@@ -1121,6 +1121,27 @@ public String toString() {
}
}
+ public static class ListSize implements Validator {
+ final int maxSize;
+
+ private ListSize(final int maxSize) {
+ this.maxSize = maxSize;
+ }
+
+ public static ListSize max(final int maxSize) {
Review comment:
Could you rename this method to somthing a bit more specific like `atMostOfLength()`?
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.ThreadMetadata;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Transformer;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestCondition;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static java.util.Arrays.asList;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+
+@Category({IntegrationTest.class})
+public class RackAwarenessIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ private static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ @Rule
+ public TestName testName = new TestName();
+
+ private static final String INPUT_TOPIC = "input-topic";
+ private static final String TAG_ZONE = "zone";
+ private static final String TAG_CLUSTER = "cluster";
+
+ private List<KafkaStreamsWithConfiguration> kafkaStreamsInstances;
+ private Properties baseConfiguration;
+
+ @BeforeClass
+ public static void createTopics() throws Exception {
+ CLUSTER.start();
+ CLUSTER.createTopic(INPUT_TOPIC, 6, 1);
+ }
+
+ @Before
+ public void setup() {
+ kafkaStreamsInstances = new ArrayList<>();
+ baseConfiguration = new Properties();
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+ final String applicationId = "app-" + safeTestName;
+ baseConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ baseConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ baseConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ baseConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ baseConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ }
+
+ @After
+ public void cleanup() throws IOException {
+ for (final KafkaStreamsWithConfiguration kafkaStreamsWithConfiguration : kafkaStreamsInstances) {
+ kafkaStreamsWithConfiguration.kafkaStreams.close(Duration.ofMillis(IntegrationTestUtils.DEFAULT_TIMEOUT));
+ IntegrationTestUtils.purgeLocalStreamsState(kafkaStreamsWithConfiguration.configuration);
+ }
+ kafkaStreamsInstances.clear();
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasBasedOnClientTags() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 1;
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilRackAwareTaskDistributionIsReached(TAG_ZONE);
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasOverMultipleClientTags() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 2;
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1c", "k8s-cluster-1"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-2"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-2"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1c", "k8s-cluster-2"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags("eu-central-1a", "k8s-cluster-3"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1b", "k8s-cluster-3"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags("eu-central-1c", "k8s-cluster-3"), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+
+ waitUntilRackAwareTaskDistributionIsReached(TAG_ZONE, TAG_CLUSTER);
+ }
+
Review comment:
Could you add a test where standbys cannot be distributed over all tags to check that task assignment works in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1068813143
Thanks @showuon for the feedback. Should be better now.
Also reduced test timeout to 1.5 minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1065347322
Hi @cadonna
Not yet. Will open it for review by Monday.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825763496
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/ClientTagAwareStandbyTaskAssignor.java
##########
@@ -211,17 +194,32 @@ static void assignStandbyTasksToClientsWithDifferentTags(final ConstrainedPriori
break;
}
- clientStates.get(clientOnUnusedTagDimensions).assignStandby(activeTaskId);
-
+ final ClientState clientStateOnUsedTagDimensions = clientStates.get(clientOnUnusedTagDimensions);
countOfUsedClients++;
numRemainingStandbys--;
+ log.debug("Assigning {} out of {} standby tasks for an active task [{}] with client tags {}. " +
Review comment:
This was not request during the previous PR review, but I think it can be useful for debugging to have extra logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1066811726
Thanks @cadonna for the feedback.
I've addressed your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826908752
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
I am a bit concerned about the length. I would propose the following:
```
MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 5
MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 20
MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 30
```
We can always increase these numbers in future if they are too small, but if we want to decrease them, we might run into backwards compatibility issues.
But more important, we should have an integration test that tests the maximum allowed size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1067964346
Thanks @cadonna for the review.
I've addressed your comment in this commit https://github.com/apache/kafka/pull/11837/commits/fbdb17016a3674e234f6ecdbe672472ee405f52d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825757133
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.ThreadMetadata;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Transformer;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestCondition;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static java.util.Arrays.asList;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+
+@Category({IntegrationTest.class})
+public class RackAwarenessIntegrationTest {
Review comment:
Since core logic tests are already written as unit tests, this integration tests verifies basic high level functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825762885
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/ClientTagAwareStandbyTaskAssignor.java
##########
@@ -211,17 +194,32 @@ static void assignStandbyTasksToClientsWithDifferentTags(final ConstrainedPriori
break;
}
- clientStates.get(clientOnUnusedTagDimensions).assignStandby(activeTaskId);
-
+ final ClientState clientStateOnUsedTagDimensions = clientStates.get(clientOnUnusedTagDimensions);
countOfUsedClients++;
numRemainingStandbys--;
+ log.debug("Assigning {} out of {} standby tasks for an active task [{}] with client tags {}. " +
+ "Standby task client tags are {}.",
+ numberOfStandbyClients - numRemainingStandbys, numberOfStandbyClients, activeTaskId,
+ clientStates.get(activeTaskClient).clientTags(), clientStateOnUsedTagDimensions.clientTags());
+
+ clientStateOnUsedTagDimensions.assignStandby(activeTaskId);
lastUsedClient = clientOnUnusedTagDimensions;
} while (numRemainingStandbys > 0);
if (numRemainingStandbys > 0) {
pendingStandbyTasksToClientId.put(activeTaskId, activeTaskClient);
tasksToRemainingStandbys.put(activeTaskId, numRemainingStandbys);
+ log.warn("Rack aware standby task assignment was not able to assign {} of {} standby tasks for the " +
Review comment:
Added logging based on the previous PR comment https://github.com/apache/kafka/pull/10851#discussion_r817629787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825998032
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignor.java
##########
@@ -1464,6 +1464,10 @@ protected byte uniqueField() {
return uniqueField;
}
+ protected Map<String, String> clientTags() {
+ return clientTags;
Review comment:
Sorry, my bad. I misinterpreted the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069887108
Nice work @lkokhreidze ! And nice team work, @cadonna ! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069087407
@showuon
I've reduced number of stateful tasks to 9. This should eliminate the flakiness.
I will do the system test in the follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069347833
FYI, if I'm looking at test results correctly, seems like latest failures are not related.
```
Build / JDK 8 and Scala 2.12 / shouldRestoreState – org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest
1m 3s
Build / JDK 17 and Scala 2.13 / [1] Type=Raft, Name=testElectionResultOutput, Security=PLAINTEXT – kafka.admin.LeaderElectionCommandTest
1m 10s
Build / JDK 17 and Scala 2.13 / shouldRestoreState – org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest
1m 2s
Build / JDK 17 and Scala 2.13 / shouldRestoreState – org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest
1m 3s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827059629
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
Apologies, my mistake.
Makes sense. I've added `RackAwarenessIntegrationTest#shouldDoRebalancingWithMaximumNumberOfClientTags` that verifies rebalancing, as well as that rack awareness works after rebalancing too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827955917
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
If after your PR the tests are still flaky, let's just test with 6 tasks and merge this PR. You can then add a system test for the scenario with larger subscriptions in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825755024
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
Do not have any since behind those restrictions, but thought it's better to have something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826097248
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignor.java
##########
@@ -1464,6 +1464,10 @@ protected byte uniqueField() {
return uniqueField;
}
+ protected Map<String, String> clientTags() {
+ return clientTags;
Review comment:
It is indeed only used in tests. But seems like that's a common pattern in the `StreamsPartitionAssignor`. Followed the same approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1066575752
Ready for the review @cadonna @showuon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826671373
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
> Actually, I don't think we need this limitation. Could we not be able to support list size > 10, key len > 50, value len > 100?
I think we need some limitations because this information will be encoded into the subscription info and subscription info is the part of the group metadata message that we need to write into the internal topic. This was also discussed in the voting thread.
As for why those numbers, I really can't say. It can be more, can be less. But we need some limitation I think. But I can't imagine valid use case of having more than 10 tags with more than 50 and 100 characters length.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825994022
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignor.java
##########
@@ -1464,6 +1464,10 @@ protected byte uniqueField() {
return uniqueField;
}
+ protected Map<String, String> clientTags() {
+ return clientTags;
Review comment:
This seems to be only used in its own test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1066647730
There are few tests failing. Will be fixing them now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825762550
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/StandbyTaskAssignmentUtils.java
##########
@@ -49,7 +53,11 @@ static int pollClientAndMaybeAssignAndUpdateRemainingStandbyTasks(final Map<UUID
tasksToRemainingStandbys.put(activeTaskId, numRemainingStandbys);
}
- return numRemainingStandbys;
+ log.warn("Unable to assign {} of {} standby tasks for task [{}]. " +
Review comment:
Refactored based on previous PR comment
https://github.com/apache/kafka/pull/10851#discussion_r817629787
Since both standby task assignors have the same logging, moved the logging to this utility method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069076197
@lkokhreidze , let's make sure the tests stable before we merge this final piece of PRs. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826766945
##########
File path: clients/src/test/java/org/apache/kafka/common/config/ConfigDefTest.java
##########
@@ -722,4 +724,35 @@ public void testNiceTimeUnits() {
assertEquals(" (365 days)", ConfigDef.niceTimeUnits(Duration.ofDays(365).toMillis()));
}
+ @Test
+ public void testThrowsExceptionWhenListSizeExceedsLimit() {
+ assertThrows(ConfigException.class, () -> new ConfigDef().define("lst",
+ Type.LIST,
+ asList("a", "b"),
+ ListSize.atMostOfSize(1),
+ Importance.HIGH,
+ "lst doc"));
+ }
+
+ @Test
+ public void testNoExceptionIsThrownWhenListSizeEqualsTheLimit() {
+ final List<String> lst = asList("a", "b", "c");
+ new ConfigDef().define("lst",
Review comment:
We can use `assertDoesNotThrow` here and below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826762433
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/StandbyTaskAssignmentUtils.java
##########
@@ -49,7 +53,11 @@ static int pollClientAndMaybeAssignAndUpdateRemainingStandbyTasks(final Map<UUID
tasksToRemainingStandbys.put(activeTaskId, numRemainingStandbys);
}
- return numRemainingStandbys;
+ log.warn("Unable to assign {} of {} standby tasks for task [{}]. " +
Review comment:
This doesn't look right. We log the warning to user even when `numRemainingStandbys==0`?
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/StandbyTaskAssignmentUtils.java
##########
@@ -26,17 +27,20 @@
import static java.util.stream.Collectors.toMap;
final class StandbyTaskAssignmentUtils {
- private StandbyTaskAssignmentUtils() {}
+ private StandbyTaskAssignmentUtils() {
+ }
Review comment:
nit: `{}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069360043
Big thank you @cadonna and @showuon for all the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827786084
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
Pushed changes to reduce number of stateful tasks to 100
https://github.com/apache/kafka/pull/11837/commits/e6f717229acb4850aec1f33e13cd88cdc3ccb4bb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069103952
@lkokhreidze We also need some updates to the docs. The sections to update are https://kafka.apache.org/31/documentation/streams/architecture#streams_architecture_recovery and maybe also https://kafka.apache.org/31/documentation/streams/developer-guide/config-streams.html.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825755435
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -1146,6 +1187,36 @@ private void verifyEOSTransactionTimeoutCompatibility() {
configUpdates.put(COMMIT_INTERVAL_MS_CONFIG, EOS_DEFAULT_COMMIT_INTERVAL_MS);
}
+ // validate rack awareness related configuration
+ final List<String> rackAwareAssignmentTags = getList(RACK_AWARE_ASSIGNMENT_TAGS_CONFIG);
Review comment:
Not sure if this is the best place to perform those validations, but couldn't find any better place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826832196
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/StandbyTaskAssignmentUtils.java
##########
@@ -49,7 +53,11 @@ static int pollClientAndMaybeAssignAndUpdateRemainingStandbyTasks(final Map<UUID
tasksToRemainingStandbys.put(activeTaskId, numRemainingStandbys);
}
- return numRemainingStandbys;
+ log.warn("Unable to assign {} of {} standby tasks for task [{}]. " +
Review comment:
Stupid mistake. Fixed. Also added tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826649424
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.ThreadMetadata;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Transformer;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestCondition;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.function.Predicate;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+
+@Category({IntegrationTest.class})
+public class RackAwarenessIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ private static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private static final String TAG_VALUE_K8_CLUSTER_1 = "k8s-cluster-1";
+ private static final String TAG_VALUE_K8_CLUSTER_2 = "k8s-cluster-2";
+ private static final String TAG_VALUE_K8_CLUSTER_3 = "k8s-cluster-3";
+ private static final String TAG_VALUE_EU_CENTRAL_1A = "eu-central-1a";
+ private static final String TAG_VALUE_EU_CENTRAL_1B = "eu-central-1b";
+ private static final String TAG_VALUE_EU_CENTRAL_1C = "eu-central-1c";
+
+ @Rule
+ public TestName testName = new TestName();
+
+ private static final String INPUT_TOPIC = "input-topic";
+ private static final String TAG_ZONE = "zone";
+ private static final String TAG_CLUSTER = "cluster";
+
+ private List<KafkaStreamsWithConfiguration> kafkaStreamsInstances;
+ private Properties baseConfiguration;
+
+ @BeforeClass
+ public static void createTopics() throws Exception {
+ CLUSTER.start();
+ CLUSTER.createTopic(INPUT_TOPIC, 6, 1);
+ }
+
+ @Before
+ public void setup() {
+ kafkaStreamsInstances = new ArrayList<>();
+ baseConfiguration = new Properties();
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+ final String applicationId = "app-" + safeTestName;
+ baseConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ baseConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ baseConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ baseConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ baseConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ }
+
+ @After
+ public void cleanup() throws IOException {
+ for (final KafkaStreamsWithConfiguration kafkaStreamsWithConfiguration : kafkaStreamsInstances) {
+ kafkaStreamsWithConfiguration.kafkaStreams.close(Duration.ofMillis(IntegrationTestUtils.DEFAULT_TIMEOUT));
+ IntegrationTestUtils.purgeLocalStreamsState(kafkaStreamsWithConfiguration.configuration);
+ }
+ kafkaStreamsInstances.clear();
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasWhenAllClientsAreLocatedOnASameClusterTag() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 1;
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(singletonList(TAG_ZONE));
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasOverMultipleClientTags() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 2;
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(asList(TAG_ZONE, TAG_CLUSTER));
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasWhenIdealDistributionCanNotBeAchieved() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 2;
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(singletonList(TAG_ZONE));
+ waitUntilPartialTaskDistributionIsReachedForTags(singletonList(TAG_CLUSTER));
+ }
+
+ private void waitUntilAllKafkaStreamsClientsAreRunning() throws Exception {
+ TestUtils.waitForCondition(
+ () -> kafkaStreamsInstances.stream().allMatch(it -> KafkaStreams.State.RUNNING == it.kafkaStreams.state()),
+ IntegrationTestUtils.DEFAULT_TIMEOUT,
+ String.format("Kafka Streams did not transit to state %s in %d milliseconds",
+ KafkaStreams.State.RUNNING, IntegrationTestUtils.DEFAULT_TIMEOUT)
+ );
+ }
Review comment:
We already have `startApplicationAndWaitUntilRunning` or `waitForApplicationState` in `IntegrationTestUtils`. I think we can use `startApplicationAndWaitUntilRunning` in our tests.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.ThreadMetadata;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Transformer;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestCondition;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.function.Predicate;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+
+@Category({IntegrationTest.class})
+public class RackAwarenessIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ private static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private static final String TAG_VALUE_K8_CLUSTER_1 = "k8s-cluster-1";
+ private static final String TAG_VALUE_K8_CLUSTER_2 = "k8s-cluster-2";
+ private static final String TAG_VALUE_K8_CLUSTER_3 = "k8s-cluster-3";
+ private static final String TAG_VALUE_EU_CENTRAL_1A = "eu-central-1a";
+ private static final String TAG_VALUE_EU_CENTRAL_1B = "eu-central-1b";
+ private static final String TAG_VALUE_EU_CENTRAL_1C = "eu-central-1c";
+
+ @Rule
+ public TestName testName = new TestName();
+
+ private static final String INPUT_TOPIC = "input-topic";
+ private static final String TAG_ZONE = "zone";
+ private static final String TAG_CLUSTER = "cluster";
+
+ private List<KafkaStreamsWithConfiguration> kafkaStreamsInstances;
+ private Properties baseConfiguration;
+
+ @BeforeClass
+ public static void createTopics() throws Exception {
+ CLUSTER.start();
+ CLUSTER.createTopic(INPUT_TOPIC, 6, 1);
+ }
+
+ @Before
+ public void setup() {
+ kafkaStreamsInstances = new ArrayList<>();
+ baseConfiguration = new Properties();
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+ final String applicationId = "app-" + safeTestName;
+ baseConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ baseConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ baseConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ baseConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ baseConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ }
+
+ @After
+ public void cleanup() throws IOException {
+ for (final KafkaStreamsWithConfiguration kafkaStreamsWithConfiguration : kafkaStreamsInstances) {
+ kafkaStreamsWithConfiguration.kafkaStreams.close(Duration.ofMillis(IntegrationTestUtils.DEFAULT_TIMEOUT));
+ IntegrationTestUtils.purgeLocalStreamsState(kafkaStreamsWithConfiguration.configuration);
+ }
+ kafkaStreamsInstances.clear();
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasWhenAllClientsAreLocatedOnASameClusterTag() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 1;
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(singletonList(TAG_ZONE));
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasOverMultipleClientTags() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 2;
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(asList(TAG_ZONE, TAG_CLUSTER));
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasWhenIdealDistributionCanNotBeAchieved() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 2;
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(singletonList(TAG_ZONE));
+ waitUntilPartialTaskDistributionIsReachedForTags(singletonList(TAG_CLUSTER));
Review comment:
Since we've confirmed the stream clients are in running state, should we also `wait` for the tasks distribution? I think we can directly verify them, right?
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
Actually, I don't think we need this limitation. Could we not be able to support list size > 10, key len > 50, value len > 100? I guess we can. If so, why this limitation? And like what you've asked, why this number?
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -1146,6 +1187,36 @@ private void verifyEOSTransactionTimeoutCompatibility() {
configUpdates.put(COMMIT_INTERVAL_MS_CONFIG, EOS_DEFAULT_COMMIT_INTERVAL_MS);
}
+ // validate rack awareness related configuration
+ final List<String> rackAwareAssignmentTags = getList(RACK_AWARE_ASSIGNMENT_TAGS_CONFIG);
Review comment:
Maybe put the rack awareness validation into an another method. (If we decided to have these limitation)
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -1146,6 +1187,36 @@ private void verifyEOSTransactionTimeoutCompatibility() {
configUpdates.put(COMMIT_INTERVAL_MS_CONFIG, EOS_DEFAULT_COMMIT_INTERVAL_MS);
}
+ // validate rack awareness related configuration
+ final List<String> rackAwareAssignmentTags = getList(RACK_AWARE_ASSIGNMENT_TAGS_CONFIG);
Review comment:
Yes, put validation here is good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069115152
Yes, doing it in a follow-up PR is totally fine!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827777681
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
Sounds good to me. Will push changes with 100 stateful tasks for the rebalancing test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825755024
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
Do not have any science behind those restrictions, but thought it's better to have something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826828584
##########
File path: clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java
##########
@@ -1121,6 +1121,27 @@ public String toString() {
}
}
+ public static class ListSize implements Validator {
+ final int maxSize;
+
+ private ListSize(final int maxSize) {
+ this.maxSize = maxSize;
+ }
+
+ public static ListSize atMostOfSize(final int maxSize) {
+ return new ListSize(maxSize);
+ }
+
+ @Override
+ public void ensureValid(final String name, final Object value) {
+ @SuppressWarnings("unchecked")
+ List<String> values = (List<String>) value;
+ if (values.size() > maxSize) {
+ throw new ConfigException(name, value, "exceeds maximum list size of [" + maxSize + "].");
Review comment:
It should output list of strings. Added test to verify.
##########
File path: clients/src/test/java/org/apache/kafka/common/config/ConfigDefTest.java
##########
@@ -722,4 +724,35 @@ public void testNiceTimeUnits() {
assertEquals(" (365 days)", ConfigDef.niceTimeUnits(Duration.ofDays(365).toMillis()));
}
+ @Test
+ public void testThrowsExceptionWhenListSizeExceedsLimit() {
+ assertThrows(ConfigException.class, () -> new ConfigDef().define("lst",
+ Type.LIST,
+ asList("a", "b"),
+ ListSize.atMostOfSize(1),
+ Importance.HIGH,
+ "lst doc"));
+ }
+
+ @Test
+ public void testNoExceptionIsThrownWhenListSizeEqualsTheLimit() {
+ final List<String> lst = asList("a", "b", "c");
+ new ConfigDef().define("lst",
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1065339340
@lkokhreidze Is this PR ready for review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826672891
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.ThreadMetadata;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Transformer;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestCondition;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.function.Predicate;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+
+@Category({IntegrationTest.class})
+public class RackAwarenessIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ private static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private static final String TAG_VALUE_K8_CLUSTER_1 = "k8s-cluster-1";
+ private static final String TAG_VALUE_K8_CLUSTER_2 = "k8s-cluster-2";
+ private static final String TAG_VALUE_K8_CLUSTER_3 = "k8s-cluster-3";
+ private static final String TAG_VALUE_EU_CENTRAL_1A = "eu-central-1a";
+ private static final String TAG_VALUE_EU_CENTRAL_1B = "eu-central-1b";
+ private static final String TAG_VALUE_EU_CENTRAL_1C = "eu-central-1c";
+
+ @Rule
+ public TestName testName = new TestName();
+
+ private static final String INPUT_TOPIC = "input-topic";
+ private static final String TAG_ZONE = "zone";
+ private static final String TAG_CLUSTER = "cluster";
+
+ private List<KafkaStreamsWithConfiguration> kafkaStreamsInstances;
+ private Properties baseConfiguration;
+
+ @BeforeClass
+ public static void createTopics() throws Exception {
+ CLUSTER.start();
+ CLUSTER.createTopic(INPUT_TOPIC, 6, 1);
+ }
+
+ @Before
+ public void setup() {
+ kafkaStreamsInstances = new ArrayList<>();
+ baseConfiguration = new Properties();
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+ final String applicationId = "app-" + safeTestName;
+ baseConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ baseConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ baseConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ baseConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ baseConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ }
+
+ @After
+ public void cleanup() throws IOException {
+ for (final KafkaStreamsWithConfiguration kafkaStreamsWithConfiguration : kafkaStreamsInstances) {
+ kafkaStreamsWithConfiguration.kafkaStreams.close(Duration.ofMillis(IntegrationTestUtils.DEFAULT_TIMEOUT));
+ IntegrationTestUtils.purgeLocalStreamsState(kafkaStreamsWithConfiguration.configuration);
+ }
+ kafkaStreamsInstances.clear();
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasWhenAllClientsAreLocatedOnASameClusterTag() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 1;
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(singletonList(TAG_ZONE));
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasOverMultipleClientTags() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 2;
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_3), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(asList(TAG_ZONE, TAG_CLUSTER));
+ }
+
+ @Test
+ public void shouldDistributeStandbyReplicasWhenIdealDistributionCanNotBeAchieved() throws Exception {
+ final Topology topology = createStatefulTopology();
+ final int numberOfStandbyReplicas = 2;
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_1), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1A, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1B, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+ createAndStart(topology, buildClientTags(TAG_VALUE_EU_CENTRAL_1C, TAG_VALUE_K8_CLUSTER_2), asList(TAG_ZONE, TAG_CLUSTER), numberOfStandbyReplicas);
+
+ waitUntilAllKafkaStreamsClientsAreRunning();
+ waitUntilIdealTaskDistributionIsReachedForTags(singletonList(TAG_ZONE));
+ waitUntilPartialTaskDistributionIsReachedForTags(singletonList(TAG_CLUSTER));
Review comment:
Yes good point. Will update the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1067839260
Hi @showuon
Thanks for taking a look. I've addressed your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827002615
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
I think we misunderstood each other. I was asking for an integration test that verifies that the rebalancing work for the maximum length of the tags. To see if the subscription has still a size that works. Maybe also in combination with many tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827083107
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
> Maybe also in combination with many tasks.
FYI - current topology generates 6 tasks, I guess that should be enough?
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
> Maybe also in combination with many tasks.
FYI - current topology generates 6 tasks, I guess that should be enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827942693
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
Even with 100 tasks tests on CI failed with timeout. I've bumped timeout to 1.5 minutes in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna edited a comment on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna edited a comment on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1069103952
@lkokhreidze We also need some updates to the docs. The sections to update are https://kafka.apache.org/31/documentation/streams/architecture#streams_architecture_recovery and maybe also https://kafka.apache.org/31/documentation/streams/developer-guide/config-streams.html. Not in this PR, please a new PR for the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827002615
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
I think we misunderstood each other. I was asking for an integration test that verifies that the rebalancing works for the maximum length of the tags. To see if the subscription has still a size that works. Maybe also in combination with many tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827603227
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/ClientTagAwareStandbyTaskAssignor.java
##########
@@ -60,7 +60,7 @@ public boolean assign(final Map<UUID, ClientState> clients,
final Map<TaskId, Integer> tasksToRemainingStandbys = computeTasksToRemainingStandbys(
numStandbyReplicas,
- allTaskIds
+ statefulTaskIds
Review comment:
nice catch, and thanks for adding a test for it.
##########
File path: streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/StandbyTaskAssignmentUtilsTest.java
##########
@@ -55,38 +65,38 @@ public void setup() {
@Test
public void shouldReturnNumberOfStandbyTasksThatWereNotAssigned() {
- final Map<TaskId, Integer> tasksToRemainingStandbys = computeTasksToRemainingStandbys(3, ACTIVE_TASKS);
-
- assertTrue(tasksToRemainingStandbys.keySet()
- .stream()
- .map(taskId -> pollClientAndMaybeAssignAndUpdateRemainingStandbyTasks(
- clients,
- tasksToRemainingStandbys,
- clientsByTaskLoad,
- taskId
- ))
- .allMatch(numRemainingStandbys -> numRemainingStandbys == 1));
+ final Logger logMock = mock(Logger.class);
+ final int numStandbyReplicas = 3;
+ final Map<TaskId, Integer> tasksToRemainingStandbys = computeTasksToRemainingStandbys(numStandbyReplicas, ACTIVE_TASKS);
+
+ tasksToRemainingStandbys.keySet().forEach(taskId -> pollClientAndMaybeAssignAndUpdateRemainingStandbyTasks(numStandbyReplicas,
+ clients,
+ tasksToRemainingStandbys,
+ clientsByTaskLoad,
+ taskId,
+ logMock));
assertTrue(ACTIVE_TASKS.stream().allMatch(activeTask -> tasksToRemainingStandbys.get(activeTask) == 1));
assertTrue(areStandbyTasksPresentForAllActiveTasks(2));
+ verify(logMock, times(ACTIVE_TASKS.size())).warn(anyString(), anyInt(), anyInt(), any());
}
@Test
public void shouldReturnZeroWhenAllStandbyTasksWereSuccessfullyAssigned() {
- final Map<TaskId, Integer> tasksToRemainingStandbys = computeTasksToRemainingStandbys(1, ACTIVE_TASKS);
-
- assertTrue(tasksToRemainingStandbys.keySet()
- .stream()
- .map(taskId -> pollClientAndMaybeAssignAndUpdateRemainingStandbyTasks(
- clients,
- tasksToRemainingStandbys,
- clientsByTaskLoad,
- taskId
- ))
- .allMatch(numRemainingStandbys -> numRemainingStandbys == 0));
+ final Logger logMock = mock(Logger.class);
+ final int numStandbyReplicas = 1;
+ final Map<TaskId, Integer> tasksToRemainingStandbys = computeTasksToRemainingStandbys(numStandbyReplicas, ACTIVE_TASKS);
+
+ tasksToRemainingStandbys.keySet().forEach(taskId -> pollClientAndMaybeAssignAndUpdateRemainingStandbyTasks(numStandbyReplicas,
+ clients,
+ tasksToRemainingStandbys,
+ clientsByTaskLoad,
+ taskId,
+ logMock));
assertTrue(ACTIVE_TASKS.stream().allMatch(activeTask -> tasksToRemainingStandbys.get(activeTask) == 0));
assertTrue(areStandbyTasksPresentForAllActiveTasks(1));
+ verifyNoInteractions(logMock);
Review comment:
Thanks for the tests
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.ThreadMetadata;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Repartitioned;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@Category({IntegrationTest.class})
+public class RackAwarenessIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ private static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private static final String TAG_VALUE_K8_CLUSTER_1 = "k8s-cluster-1";
+ private static final String TAG_VALUE_K8_CLUSTER_2 = "k8s-cluster-2";
+ private static final String TAG_VALUE_K8_CLUSTER_3 = "k8s-cluster-3";
+ private static final String TAG_VALUE_EU_CENTRAL_1A = "eu-central-1a";
+ private static final String TAG_VALUE_EU_CENTRAL_1B = "eu-central-1b";
+ private static final String TAG_VALUE_EU_CENTRAL_1C = "eu-central-1c";
+
+ private static final int DEFAULT_NUMBER_OF_STATEFUL_SUB_TOPOLOGIES = 1;
+ private static final int DEFAULT_FANOUT_NUMBER_OF_PARTITIONS = 2;
+
+ @Rule
+ public TestName testName = new TestName();
+
+ private static final String INPUT_TOPIC = "input-topic";
+
+ private static final String TAG_ZONE = "zone";
+ private static final String TAG_CLUSTER = "cluster";
+
+ private List<KafkaStreamsWithConfiguration> kafkaStreamsInstances;
+ private Properties baseConfiguration;
+ private Topology topology;
+
+ @BeforeClass
+ public static void createTopics() throws Exception {
+ CLUSTER.start();
+ CLUSTER.createTopic(INPUT_TOPIC, 6, 1);
+ }
+
+ @Before
+ public void setup() {
+ kafkaStreamsInstances = new ArrayList<>();
+ baseConfiguration = new Properties();
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+ final String applicationId = "app-" + safeTestName;
+ baseConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ baseConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ baseConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ baseConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ baseConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ }
+
+ @After
+ public void cleanup() throws IOException {
+ for (final KafkaStreamsWithConfiguration kafkaStreamsWithConfiguration : kafkaStreamsInstances) {
+ kafkaStreamsWithConfiguration.kafkaStreams.close(Duration.ofMillis(IntegrationTestUtils.DEFAULT_TIMEOUT));
+ IntegrationTestUtils.purgeLocalStreamsState(kafkaStreamsWithConfiguration.configuration);
+ }
+ kafkaStreamsInstances.clear();
+ }
+
+ @Test
+ public void shouldDoRebalancingWithMaximumNumberOfClientTags() throws Exception {
+ initTopology(100, 2);
+ final Duration timeout = Duration.ofMinutes(3);
Review comment:
3 mins for a test is too long IMO. Could we reduce to 2 mins or less?
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
@cadonna , Good suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827776021
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
I see. I asked some other people and apparently the size of the subscription is not that much of a concern currently. I would still propose to keep the test with 100 instead of 200 tasks. With 200 tasks, the test takes ~10 s on my laptop whereas with 100 tasks it takes ~6 s. I think 6 s is acceptable and might also reduce flakiness. @lkokhreidze and @showuon WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna merged pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna merged pull request #11837:
URL: https://github.com/apache/kafka/pull/11837
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r827098020
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/RackAwarenessIntegrationTest.java
##########
@@ -108,55 +117,122 @@ public void cleanup() throws IOException {
kafkaStreamsInstances.clear();
}
+ @Test
+ public void shouldThrowConfigExceptionWhenRackAwareAssignmentTagsExceedTheLimit() {
Review comment:
I was more thinking about 1000 stateful tasks, because the task offset sums might also inflate the subscription.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826683733
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -144,10 +148,15 @@
private static final long DEFAULT_COMMIT_INTERVAL_MS = 30000L;
private static final long EOS_DEFAULT_COMMIT_INTERVAL_MS = 100L;
private static final int DEFAULT_TRANSACTION_TIMEOUT = 10000;
+ private static final short DEFAULT_MAX_CLIENT_TAG_KEY_VALUE_LENGTH = 40;
public static final int DUMMY_THREAD_INDEX = 1;
public static final long MAX_TASK_IDLE_MS_DISABLED = -1;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_LIST_SIZE = 10;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_KEY_LENGTH = 50;
+ public static final int MAX_RACK_AWARE_ASSIGNMENT_TAG_VALUE_LENGTH = 100;
Review comment:
Make sense to me. Could we add a comment above these line, to mention we have these limitation because these tag value will be encoded into subscription info, sth like that? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825761673
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/HighAvailabilityTaskAssignor.java
##########
@@ -142,15 +142,6 @@ private void assignStandbyReplicaTasks(final TreeMap<UUID, ClientState> clientSt
);
}
- // Visible for testing
- static StandbyTaskAssignor createStandbyTaskAssignor(final AssignmentConfigs configs) {
Review comment:
Refactored based on the previous PR comment https://github.com/apache/kafka/pull/10851#discussion_r817650626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r825763496
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/ClientTagAwareStandbyTaskAssignor.java
##########
@@ -211,17 +194,32 @@ static void assignStandbyTasksToClientsWithDifferentTags(final ConstrainedPriori
break;
}
- clientStates.get(clientOnUnusedTagDimensions).assignStandby(activeTaskId);
-
+ final ClientState clientStateOnUsedTagDimensions = clientStates.get(clientOnUnusedTagDimensions);
countOfUsedClients++;
numRemainingStandbys--;
+ log.debug("Assigning {} out of {} standby tasks for an active task [{}] with client tags {}. " +
Review comment:
This was not requested during the previous PR review, but I think it can be useful for debugging to have extra logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#discussion_r826097248
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignor.java
##########
@@ -1464,6 +1464,10 @@ protected byte uniqueField() {
return uniqueField;
}
+ protected Map<String, String> clientTags() {
+ return clientTags;
Review comment:
It is indeed only used in tests. But seems like that's a common pattern in the `StreamsPartitionAssignor`. Followed the same approach. Happy to remove this if you think it's unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #11837: KAFKA-6718 / Add rack awareness configurations to StreamsConfig
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #11837:
URL: https://github.com/apache/kafka/pull/11837#issuecomment-1066982773
Ran `./gradlew :streams:testAll` locally. The only test that failed was `EosIntegrationTest. shouldCommitCorrectOffsetIfInputTopicIsTransactional[at_least_once]` but seems like a flaky test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org