You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/09/01 09:38:14 UTC
[GitHub] [kafka] lkokhreidze opened a new pull request #9237: KAFKA-10454 / integration test
lkokhreidze opened a new pull request #9237:
URL: https://github.com/apache/kafka/pull/9237
Integration test for [KAFKA-10454](https://github.com/lkokhreidze/kafka/pull/new/KAFKA-10454) bug
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-703803731
@lkokhreidze, thanks for the quick update. I'll make another pass soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r485885898
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -632,8 +632,18 @@ public final void addInternalTopic(final String topicName,
internalTopicNamesWithProperties.put(topicName, internalTopicProperties);
}
- public final void copartitionSources(final Collection<String> sourceNodes) {
- copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
+ public final synchronized void copartitionSources(final Collection<String> sourceNodes) {
+ copartitionSourceGroups.add(new HashSet<>(sourceNodes));
+ }
+
+ public final synchronized void maybeUpdateCopartitionSourceGroups(final String replacedNodeName,
Review comment:
I'm not really sure what the threading model is when building the topology, but chose to be safe and made all the accessors synchronized.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck merged pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck merged pull request #9237:
URL: https://github.com/apache/kafka/pull/9237
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-708635800
Java 8 failed [StreamTableJoinTopologyOptimizationIntegrationTest/](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-9237/17/testReport/junit/org.apache.kafka.streams.integration/StreamTableJoinTopologyOptimizationIntegrationTest/Build___JDK_8___shouldDoStreamTableJoinWithDifferentNumberOfPartitions_Optimization___none_/?cloudbees-analytics-link=scm-reporting%2Ftests%2Ffailed) seems like a flaky test failure as it was due directory clean-up.
Java 11 and Java 15 passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-713351644
@bbejeck PR https://github.com/apache/kafka/pull/9468 for 2.6
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-713180604
Thanks for the contribution @lkokhreidze!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r499692083
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -633,10 +633,20 @@ public final void addInternalTopic(final String topicName,
}
public final void copartitionSources(final Collection<String> sourceNodes) {
- copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
+ copartitionSourceGroups.add(new HashSet<>(sourceNodes));
}
- public void validateCopartition() {
+ public final void maybeUpdateCopartitionSourceGroups(final String replacedNodeName,
+ final String optimizedNodeName) {
+ for (final Set<String> copartitionSourceGroup : copartitionSourceGroups) {
+ if (copartitionSourceGroup.contains(replacedNodeName)) {
+ copartitionSourceGroup.remove(replacedNodeName);
+ copartitionSourceGroup.add(optimizedNodeName);
+ }
+ }
+ }
+
+ public synchronized void validateCopartition() {
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / integration test
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r485885898
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -632,8 +632,18 @@ public final void addInternalTopic(final String topicName,
internalTopicNamesWithProperties.put(topicName, internalTopicProperties);
}
- public final void copartitionSources(final Collection<String> sourceNodes) {
- copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
+ public final synchronized void copartitionSources(final Collection<String> sourceNodes) {
+ copartitionSourceGroups.add(new HashSet<>(sourceNodes));
+ }
+
+ public final synchronized void maybeUpdateCopartitionSourceGroups(final String replacedNodeName,
Review comment:
I'm not really sure what's the threading model is when building the topology, but chose to be safe and made both accessors synchronized.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r499670799
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -633,10 +633,20 @@ public final void addInternalTopic(final String topicName,
}
public final void copartitionSources(final Collection<String> sourceNodes) {
- copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
+ copartitionSourceGroups.add(new HashSet<>(sourceNodes));
}
- public void validateCopartition() {
+ public final void maybeUpdateCopartitionSourceGroups(final String replacedNodeName,
+ final String optimizedNodeName) {
+ for (final Set<String> copartitionSourceGroup : copartitionSourceGroups) {
+ if (copartitionSourceGroup.contains(replacedNodeName)) {
+ copartitionSourceGroup.remove(replacedNodeName);
+ copartitionSourceGroup.add(optimizedNodeName);
+ }
+ }
+ }
+
+ public synchronized void validateCopartition() {
Review comment:
nit: I think we can remove `synchronized` here as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-704347684
Java 8 failed with `kafka.network.ConnectionQuotasTest.testNoConnectionLimitsByDefault`
Java 11 failed with `org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta`
Java 15 passed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-702185633
java 8 failed with `kafka.api.PlaintextConsumerTest.testLowMaxFetchSizeForRequestAndPartition`, `org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta`
java 11 failed with `kafka.api.PlaintextAdminIntegrationTest.testAlterReplicaLogDirs`
java 15 failed with `kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup`, and `org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta`
Jira tickets updated
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-694871160
hey @vvcephei @bbejeck any updates around this? would be great to make it for the next bug release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / integration test
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r485886423
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StreamTableJoinTopologyOptimizationIntegrationTest.java
##########
@@ -0,0 +1,242 @@
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(value = Parameterized.class)
+@Category({IntegrationTest.class})
+public class StreamTableJoinTopologyOptimizationIntegrationTest {
Review comment:
There's already another `StreamTableIntegrationTest` present, but it works to `TopologyTestDriver` so I thought it would be better and easier to keep them separate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r497846379
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -632,11 +632,21 @@ public final void addInternalTopic(final String topicName,
internalTopicNamesWithProperties.put(topicName, internalTopicProperties);
}
- public final void copartitionSources(final Collection<String> sourceNodes) {
- copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
+ public final synchronized void copartitionSources(final Collection<String> sourceNodes) {
Review comment:
building a topology is done on the main thread when calling `StreamBuilder.build()` so I think it's safe to remove `synchronized`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-707945635
Hi @bbejeck @vvcephei
Sorry for pinging again, but can this PR be merged?
Would love to get it live for next releases.
Thansks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r485885898
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -632,8 +632,18 @@ public final void addInternalTopic(final String topicName,
internalTopicNamesWithProperties.put(topicName, internalTopicProperties);
}
- public final void copartitionSources(final Collection<String> sourceNodes) {
- copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
+ public final synchronized void copartitionSources(final Collection<String> sourceNodes) {
+ copartitionSourceGroups.add(new HashSet<>(sourceNodes));
+ }
+
+ public final synchronized void maybeUpdateCopartitionSourceGroups(final String replacedNodeName,
Review comment:
I'm not really sure what the threading model is when building the topology, but chose to be safe and made both accessors synchronized.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r500023844
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StreamTableJoinTopologyOptimizationIntegrationTest.java
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(value = Parameterized.class)
+@Category({IntegrationTest.class})
+public class StreamTableJoinTopologyOptimizationIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ @ClassRule
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private String tableTopic;
+ private String inputTopic;
+ private String outputTopic;
+ private String applicationId;
+
+ private Properties streamsConfiguration;
+
+ @Rule
+ public TestName testName = new TestName();
+
+ @Parameterized.Parameter
+ public String topologyOptimization;
+
+ @Parameterized.Parameters(name = "Optimization = {0}")
+ public static Collection<?> topologyOptimization() {
+ return Arrays.asList(new String[][]{
+ {StreamsConfig.OPTIMIZE},
+ {StreamsConfig.NO_OPTIMIZATION}
+ });
+ }
+
+ @Before
+ public void before() throws InterruptedException {
+ streamsConfiguration = new Properties();
+
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+
+ tableTopic = "table-topic" + safeTestName;
+ inputTopic = "stream-topic-" + safeTestName;
+ outputTopic = "output-topic-" + safeTestName;
+ applicationId = "app-" + safeTestName;
+
+ CLUSTER.createTopic(inputTopic, 4, 1);
+ CLUSTER.createTopic(tableTopic, 2, 1);
+ CLUSTER.createTopic(outputTopic, 4, 1);
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, topologyOptimization);
+ }
+
+ @After
+ public void whenShuttingDown() throws IOException {
+ IntegrationTestUtils.purgeLocalStreamsState(streamsConfiguration);
+ }
+
+ @Test
+ public void shouldDoStreamTableJoinWithDifferentNumberOfPartitions() throws Exception {
+ final String storeName = "store";
+ final String selectKeyName = "selectKey";
+
+ final StreamsBuilder streamsBuilder = new StreamsBuilder();
+
+ final KStream<Integer, String> stream = streamsBuilder.stream(inputTopic);
+ final KTable<Integer, String> table = streamsBuilder.table(tableTopic, Materialized.as(storeName));
+
+ stream
+ .selectKey((key, value) -> key, Named.as(selectKeyName))
+ .join(table, (value1, value2) -> value2)
+ .to(outputTopic);
+
+ startStreams(streamsBuilder);
+
+ final long timestamp = System.currentTimeMillis();
+
+ final List<KeyValue<Integer, String>> expectedRecords = Arrays.asList(
+ new KeyValue<>(1, "A"),
+ new KeyValue<>(2, "B")
+ );
+
+ sendEvents(inputTopic, timestamp, expectedRecords);
+ sendEvents(outputTopic, timestamp, expectedRecords);
+
+ startStreams(streamsBuilder);
+
+ validateReceivedMessages(
+ outputTopic,
+ new IntegerDeserializer(),
+ new StringDeserializer(),
+ expectedRecords
+ );
+
+ final Set<String> allTopicsInCluster = CLUSTER.getAllTopicsInCluster();
+
+ final String repartitionTopicName = applicationId + "-" + selectKeyName + "-repartition";
+ final String tableChangelogStoreName = applicationId + "-" + storeName + "-changelog";
+
+ assertTrue(topicExists(repartitionTopicName));
+ assertEquals(2, getNumberOfPartitionsForTopic(repartitionTopicName));
+
+ if (StreamsConfig.OPTIMIZE.equals(topologyOptimization)) {
+ assertFalse(allTopicsInCluster.contains(tableChangelogStoreName));
+ } else if (StreamsConfig.NO_OPTIMIZATION.equals(topologyOptimization)) {
Review comment:
I did it on purpose. Potentially there can be some changes regarding those values and felt like being explicit on what is being tested would be better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze edited a comment on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze edited a comment on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-707945635
Hi @bbejeck @vvcephei
Sorry for pinging again, but can this PR be merged?
Would love to get it live for next releases.
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-713330147
@bbejeck will do
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r499908753
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StreamTableJoinTopologyOptimizationIntegrationTest.java
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(value = Parameterized.class)
+@Category({IntegrationTest.class})
+public class StreamTableJoinTopologyOptimizationIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ @ClassRule
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private String tableTopic;
+ private String inputTopic;
+ private String outputTopic;
+ private String applicationId;
+
+ private Properties streamsConfiguration;
+
+ @Rule
+ public TestName testName = new TestName();
+
+ @Parameterized.Parameter
+ public String topologyOptimization;
+
+ @Parameterized.Parameters(name = "Optimization = {0}")
+ public static Collection<?> topologyOptimization() {
+ return Arrays.asList(new String[][]{
+ {StreamsConfig.OPTIMIZE},
+ {StreamsConfig.NO_OPTIMIZATION}
+ });
+ }
+
+ @Before
+ public void before() throws InterruptedException {
+ streamsConfiguration = new Properties();
+
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+
+ tableTopic = "table-topic" + safeTestName;
+ inputTopic = "stream-topic-" + safeTestName;
+ outputTopic = "output-topic-" + safeTestName;
+ applicationId = "app-" + safeTestName;
+
+ CLUSTER.createTopic(inputTopic, 4, 1);
+ CLUSTER.createTopic(tableTopic, 2, 1);
+ CLUSTER.createTopic(outputTopic, 4, 1);
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, topologyOptimization);
+ }
+
+ @After
+ public void whenShuttingDown() throws IOException {
+ IntegrationTestUtils.purgeLocalStreamsState(streamsConfiguration);
+ }
+
+ @Test
+ public void shouldDoStreamTableJoinWithDifferentNumberOfPartitions() throws Exception {
+ final String storeName = "store";
+ final String selectKeyName = "selectKey";
+
+ final StreamsBuilder streamsBuilder = new StreamsBuilder();
+
+ final KStream<Integer, String> stream = streamsBuilder.stream(inputTopic);
+ final KTable<Integer, String> table = streamsBuilder.table(tableTopic, Materialized.as(storeName));
+
+ stream
+ .selectKey((key, value) -> key, Named.as(selectKeyName))
+ .join(table, (value1, value2) -> value2)
+ .to(outputTopic);
+
+ startStreams(streamsBuilder);
+
+ final long timestamp = System.currentTimeMillis();
+
+ final List<KeyValue<Integer, String>> expectedRecords = Arrays.asList(
+ new KeyValue<>(1, "A"),
+ new KeyValue<>(2, "B")
+ );
+
+ sendEvents(inputTopic, timestamp, expectedRecords);
+ sendEvents(outputTopic, timestamp, expectedRecords);
+
+ startStreams(streamsBuilder);
+
+ validateReceivedMessages(
+ outputTopic,
+ new IntegerDeserializer(),
+ new StringDeserializer(),
+ expectedRecords
+ );
+
+ final Set<String> allTopicsInCluster = CLUSTER.getAllTopicsInCluster();
+
+ final String repartitionTopicName = applicationId + "-" + selectKeyName + "-repartition";
+ final String tableChangelogStoreName = applicationId + "-" + storeName + "-changelog";
+
+ assertTrue(topicExists(repartitionTopicName));
+ assertEquals(2, getNumberOfPartitionsForTopic(repartitionTopicName));
+
+ if (StreamsConfig.OPTIMIZE.equals(topologyOptimization)) {
+ assertFalse(allTopicsInCluster.contains(tableChangelogStoreName));
+ } else if (StreamsConfig.NO_OPTIMIZATION.equals(topologyOptimization)) {
Review comment:
super nit: I missed this before, but the last statement could be an `else`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-707950572
@lkokhreidze I appreciate your patience. We need to get a green build but I'm confident we'll get this in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-702185878
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r499691112
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -633,10 +633,20 @@ public final void addInternalTopic(final String topicName,
}
public final void copartitionSources(final Collection<String> sourceNodes) {
- copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
+ copartitionSourceGroups.add(new HashSet<>(sourceNodes));
}
- public void validateCopartition() {
+ public final void maybeUpdateCopartitionSourceGroups(final String replacedNodeName,
+ final String optimizedNodeName) {
+ for (final Set<String> copartitionSourceGroup : copartitionSourceGroups) {
+ if (copartitionSourceGroup.contains(replacedNodeName)) {
+ copartitionSourceGroup.remove(replacedNodeName);
+ copartitionSourceGroup.add(optimizedNodeName);
+ }
+ }
+ }
+
+ public synchronized void validateCopartition() {
Review comment:
Sorry, somehow missed this one. On it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-689804031
Ran `./gradlew :streams:clean :streams:test` locally, everything passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-702186897
Ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-713197074
@lkokhreidze I tried to cherry-pick this to 2.6.1 as well, but the `TOPOLOGY_OPITIMIZATION_CONFIG` is part of 2.7 and doesn't compile on 2.6. Can you do a separate PR targeted for the 2.6 branch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #9237: KAFKA-10454 / integration test
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-689788570
call for review @mjsax @vvcephei @guozhangwang @bbejeck
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-702190153
@lkokhreidze do you mind rebasing? There was a PR (#9238) merged on Sept 1st (same date as this PR) and I'm thinking it needs to pick up those changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-713180469
Merged #9237 into trunk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-713189428
cherry-picked to 2.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-689825230
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r500023844
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StreamTableJoinTopologyOptimizationIntegrationTest.java
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(value = Parameterized.class)
+@Category({IntegrationTest.class})
+public class StreamTableJoinTopologyOptimizationIntegrationTest {
+ private static final int NUM_BROKERS = 1;
+
+ @ClassRule
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private String tableTopic;
+ private String inputTopic;
+ private String outputTopic;
+ private String applicationId;
+
+ private Properties streamsConfiguration;
+
+ @Rule
+ public TestName testName = new TestName();
+
+ @Parameterized.Parameter
+ public String topologyOptimization;
+
+ @Parameterized.Parameters(name = "Optimization = {0}")
+ public static Collection<?> topologyOptimization() {
+ return Arrays.asList(new String[][]{
+ {StreamsConfig.OPTIMIZE},
+ {StreamsConfig.NO_OPTIMIZATION}
+ });
+ }
+
+ @Before
+ public void before() throws InterruptedException {
+ streamsConfiguration = new Properties();
+
+ final String safeTestName = safeUniqueTestName(getClass(), testName);
+
+ tableTopic = "table-topic" + safeTestName;
+ inputTopic = "stream-topic-" + safeTestName;
+ outputTopic = "output-topic-" + safeTestName;
+ applicationId = "app-" + safeTestName;
+
+ CLUSTER.createTopic(inputTopic, 4, 1);
+ CLUSTER.createTopic(tableTopic, 2, 1);
+ CLUSTER.createTopic(outputTopic, 4, 1);
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath());
+ streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Integer().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, topologyOptimization);
+ }
+
+ @After
+ public void whenShuttingDown() throws IOException {
+ IntegrationTestUtils.purgeLocalStreamsState(streamsConfiguration);
+ }
+
+ @Test
+ public void shouldDoStreamTableJoinWithDifferentNumberOfPartitions() throws Exception {
+ final String storeName = "store";
+ final String selectKeyName = "selectKey";
+
+ final StreamsBuilder streamsBuilder = new StreamsBuilder();
+
+ final KStream<Integer, String> stream = streamsBuilder.stream(inputTopic);
+ final KTable<Integer, String> table = streamsBuilder.table(tableTopic, Materialized.as(storeName));
+
+ stream
+ .selectKey((key, value) -> key, Named.as(selectKeyName))
+ .join(table, (value1, value2) -> value2)
+ .to(outputTopic);
+
+ startStreams(streamsBuilder);
+
+ final long timestamp = System.currentTimeMillis();
+
+ final List<KeyValue<Integer, String>> expectedRecords = Arrays.asList(
+ new KeyValue<>(1, "A"),
+ new KeyValue<>(2, "B")
+ );
+
+ sendEvents(inputTopic, timestamp, expectedRecords);
+ sendEvents(outputTopic, timestamp, expectedRecords);
+
+ startStreams(streamsBuilder);
+
+ validateReceivedMessages(
+ outputTopic,
+ new IntegerDeserializer(),
+ new StringDeserializer(),
+ expectedRecords
+ );
+
+ final Set<String> allTopicsInCluster = CLUSTER.getAllTopicsInCluster();
+
+ final String repartitionTopicName = applicationId + "-" + selectKeyName + "-repartition";
+ final String tableChangelogStoreName = applicationId + "-" + storeName + "-changelog";
+
+ assertTrue(topicExists(repartitionTopicName));
+ assertEquals(2, getNumberOfPartitionsForTopic(repartitionTopicName));
+
+ if (StreamsConfig.OPTIMIZE.equals(topologyOptimization)) {
+ assertFalse(allTopicsInCluster.contains(tableChangelogStoreName));
+ } else if (StreamsConfig.NO_OPTIMIZATION.equals(topologyOptimization)) {
Review comment:
I did it on purpose. Potentially there can be some changes regarding those values and felt like being explicit on what is being tested is would be better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-702254622
@bbejeck done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r498383696
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -632,11 +632,21 @@ public final void addInternalTopic(final String topicName,
internalTopicNamesWithProperties.put(topicName, internalTopicProperties);
}
- public final void copartitionSources(final Collection<String> sourceNodes) {
- copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
+ public final synchronized void copartitionSources(final Collection<String> sourceNodes) {
Review comment:
Thanks for confirming, removed synchronized.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lkokhreidze commented on a change in pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
lkokhreidze commented on a change in pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#discussion_r485886423
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StreamTableJoinTopologyOptimizationIntegrationTest.java
##########
@@ -0,0 +1,242 @@
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(value = Parameterized.class)
+@Category({IntegrationTest.class})
+public class StreamTableJoinTopologyOptimizationIntegrationTest {
Review comment:
There's already another `StreamTableIntegrationTest` present, but it works with `TopologyTestDriver` so I thought it would be better and easier to keep them separate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bbejeck commented on pull request #9237: KAFKA-10454 / Update copartitionSourceGroups when optimization algorithm is triggered
Posted by GitBox <gi...@apache.org>.
bbejeck commented on pull request #9237:
URL: https://github.com/apache/kafka/pull/9237#issuecomment-703926657
Note the tests for JDK 11 passed (https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-9237/12/?cloudbees-analytics-link=scm-reporting%2Fstage%2Fpending) but the job isn't exiting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org