You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "maksaska (via GitHub)" <gi...@apache.org> on 2024/06/06 13:46:26 UTC
[PR] IGNITE-22403 Algorithm improvement for kafka topic partition distribution [ignite-extensions]
maksaska opened a new pull request, #272:
URL: https://github.com/apache/ignite-extensions/pull/272
Distribution for partitions from kafka topic over requested threads in `AbstractKafkaToIgniteCdcStreamer` (CDC Kafka to Ignite) is not uniform.
There might be cases with unequal load on threads, that reads data from kafka, which might result in overhead/bottleneck and slow workflow in general for the whole CDC with Kafka process. The improved version takes that into account.
Initially, for uniform distribution to form, the streamer calculated the ratio between the number of partitions and the number of threads.It didn't take into account the modulus of the result, thus leading to underestimated number of partiotions one thread can take. The fixed version of the algorithm uses modulus to round up the underestimated number to pump up the load on the first threads. To make things work, the number of partitions taken by each thread is reevaulated for each thread, which allows to reduce the difference in parity and integral division in general. In other words, threads balance themself out while partitions are being distributed from the first thread to the last one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 CDC: Improve Kafka partition ranges assignment for appliers [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1668660530
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
+ * @param threadCnt Thread count.
+ * @param result List of pairs defining partition ranges for each applier thread.
Review Comment:
Done!
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
+ * @param threadCnt Thread count.
+ * @param result List of pairs defining partition ranges for each applier thread.
+ */
+ private void testKafkaPartitionsCase(int threadCnt, int partFrom, int partTo, List<IgniteBiTuple<Integer, Integer>> result) {
+ streamerCfg.setThreadCount(threadCnt);
+ streamerCfg.setKafkaPartsFrom(partFrom);
+ streamerCfg.setKafkaPartsTo(partTo);
+
+ assertEquals(result, kafkaPartitions(streamerCfg));
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 CDC: Improve Kafka partition ranges assignment for appliers [ignite-extensions]
Posted by "NSAmelchev (via GitHub)" <gi...@apache.org>.
NSAmelchev merged PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 CDC: Improve Kafka partition ranges assignment for appliers [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1668658781
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
+ * @param threadCnt Thread count.
+ * @param result List of pairs defining partition ranges for each applier thread.
+ */
+ private void testKafkaPartitionsCase(int threadCnt, int partFrom, int partTo, List<IgniteBiTuple<Integer, Integer>> result) {
Review Comment:
Renamed the variable accroding to the rules
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 Algorithm improvement for kafka topic partition distribution [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1666832214
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -153,26 +154,16 @@ protected void runAppliers() {
streamerCfg
);
- int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
- int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
- int threadCnt = streamerCfg.getThreadCount();
-
- int partPerApplier = kafkaParts / threadCnt;
-
- for (int i = 0; i < threadCnt; i++) {
- int from = i * partPerApplier;
- int to = (i + 1) * partPerApplier;
-
- if (i == threadCnt - 1)
- to = kafkaParts;
+ int counter = 0;
+ for (IgniteBiTuple<Integer, Integer> part : getKafkaPartitionsDistribution(streamerCfg)) {
KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
() -> eventsApplier(),
log,
kafkaProps,
streamerCfg.getTopic(),
- kafkaPartsFrom + from,
- kafkaPartsFrom + to,
+ part.get1(),
Review Comment:
Done
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -153,26 +154,16 @@ protected void runAppliers() {
streamerCfg
);
- int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
- int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
- int threadCnt = streamerCfg.getThreadCount();
-
- int partPerApplier = kafkaParts / threadCnt;
-
- for (int i = 0; i < threadCnt; i++) {
- int from = i * partPerApplier;
- int to = (i + 1) * partPerApplier;
-
- if (i == threadCnt - 1)
- to = kafkaParts;
+ int counter = 0;
+ for (IgniteBiTuple<Integer, Integer> part : getKafkaPartitionsDistribution(streamerCfg)) {
KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
() -> eventsApplier(),
log,
kafkaProps,
streamerCfg.getTopic(),
- kafkaPartsFrom + from,
- kafkaPartsFrom + to,
+ part.get1(),
+ part.get2(),
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 CDC: Improve Kafka partition ranges assignment for appliers [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1668659863
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
Review Comment:
Changed the javadoc message
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
+ * @param threadCnt Thread count.
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 CDC: Improve Kafka partition ranges assignment for appliers [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1668659237
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 Algorithm improvement for kafka topic partition distribution [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1666831917
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -153,26 +154,16 @@ protected void runAppliers() {
streamerCfg
);
- int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
- int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
- int threadCnt = streamerCfg.getThreadCount();
-
- int partPerApplier = kafkaParts / threadCnt;
-
- for (int i = 0; i < threadCnt; i++) {
- int from = i * partPerApplier;
- int to = (i + 1) * partPerApplier;
-
- if (i == threadCnt - 1)
- to = kafkaParts;
+ int counter = 0;
+ for (IgniteBiTuple<Integer, Integer> part : getKafkaPartitionsDistribution(streamerCfg)) {
Review Comment:
Done
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -197,6 +188,34 @@ protected void runAppliers() {
}
}
+ /**
+ * Creates Kafka partitions distribution for provided streamer configuration. Each element of the returning
+ * {@code List} is an {@link IgniteBiTuple}, which represents a kafka topic partition interval to be
+ * scanned by single thread.
+ * @param streamerCfg {@link KafkaToIgniteCdcStreamerConfiguration}.
+ * @return {@code List} of {@code IgniteBiTuple<Integer, Integer>}.
+ */
+ public static List<IgniteBiTuple<Integer, Integer>> getKafkaPartitionsDistribution(KafkaToIgniteCdcStreamerConfiguration streamerCfg) {
+ List<IgniteBiTuple<Integer, Integer>> distr = new ArrayList<>();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 Algorithm improvement for kafka topic partition distribution [ignite-extensions]
Posted by "shishkovilja (via GitHub)" <gi...@apache.org>.
shishkovilja commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1668467560
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -153,26 +154,16 @@ protected void runAppliers() {
streamerCfg
);
- int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
- int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
- int threadCnt = streamerCfg.getThreadCount();
-
- int partPerApplier = kafkaParts / threadCnt;
-
- for (int i = 0; i < threadCnt; i++) {
- int from = i * partPerApplier;
- int to = (i + 1) * partPerApplier;
-
- if (i == threadCnt - 1)
- to = kafkaParts;
+ int counter = 0;
Review Comment:
```suggestion
int cntr = 0;
```
See: https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -181,7 +172,7 @@ protected void runAppliers() {
stopped
);
- addAndStart("applier-thread-" + i, applier);
+ addAndStart("applier-thread-" + counter++, applier);
Review Comment:
```suggestion
addAndStart("applier-thread-" + cntr++, applier);
```
See: https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
+ * @param threadCnt Thread count.
+ * @param result List of pairs defining partition ranges for each applier thread.
+ */
+ private void testKafkaPartitionsCase(int threadCnt, int partFrom, int partTo, List<IgniteBiTuple<Integer, Integer>> result) {
+ streamerCfg.setThreadCount(threadCnt);
+ streamerCfg.setKafkaPartsFrom(partFrom);
+ streamerCfg.setKafkaPartsTo(partTo);
+
+ assertEquals(result, kafkaPartitions(streamerCfg));
Review Comment:
```suggestion
assertEquals(expParts, kafkaPartitions(streamerCfg));
```
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
+ * @param threadCnt Thread count.
+ * @param result List of pairs defining partition ranges for each applier thread.
Review Comment:
```suggestion
* @param expParts List of expected partition ranges.
```
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
+ * @param threadCnt Thread count.
+ * @param result List of pairs defining partition ranges for each applier thread.
+ */
+ private void testKafkaPartitionsCase(int threadCnt, int partFrom, int partTo, List<IgniteBiTuple<Integer, Integer>> result) {
Review Comment:
```suggestion
private void doTest(int threadCnt, int partFrom, int partTo, List<IgniteBiTuple<Integer, Integer>> expParts) {
```
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
+ * @param threadCnt Thread count.
Review Comment:
```suggestion
* @param threadCnt Applier threads count.
```
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
Review Comment:
Replace with empty javadoc, please
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
+public class KafkaToIgnitePartitionDistributionTest extends GridCommonAbstractTest {
+ /** Streamer config. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ /** Checks the topic partitions distribution over specified number of threads. */
+ @Test
+ public void testKafkaPartitions() {
+ testKafkaPartitionsCase(3, 0, 8, Arrays.asList(F.t(0, 3), F.t(3, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(4, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 8)));
+ testKafkaPartitionsCase(5, 0, 8, Arrays.asList(F.t(0, 2), F.t(2, 4), F.t(4, 6), F.t(6, 7), F.t(7, 8)));
+
+ testKafkaPartitionsCase(3, 3, 11, Arrays.asList(F.t(3, 6), F.t(6, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(4, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 11)));
+ testKafkaPartitionsCase(5, 3, 11, Arrays.asList(F.t(3, 5), F.t(5, 7), F.t(7, 9), F.t(9, 10), F.t(10, 11)));
+
+ testKafkaPartitionsCase(3, 1, 4, Arrays.asList(F.t(1, 2), F.t(2, 3), F.t(3, 4)));
+ }
+
+ /**
+ * Checks the topic partitions distribution over specified number of threads for specified parameters.
Review Comment:
Remove duplicate comment, we need only parameters description.
##########
modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/KafkaToIgnitePartitionDistributionTest.java:
##########
@@ -0,0 +1,43 @@
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.kafka.AbstractKafkaToIgniteCdcStreamer.kafkaPartitions;
+
+/** Tests the kafka topic partitions distribution over specified number of threads. */
Review Comment:
```suggestion
/** Tests Kafka partition ranges assignment to the multiple applier threads. */
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 CDC: Improve Kafka partition ranges assignment for appliers [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1668658059
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -153,26 +154,16 @@ protected void runAppliers() {
streamerCfg
);
- int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
- int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
- int threadCnt = streamerCfg.getThreadCount();
-
- int partPerApplier = kafkaParts / threadCnt;
-
- for (int i = 0; i < threadCnt; i++) {
- int from = i * partPerApplier;
- int to = (i + 1) * partPerApplier;
-
- if (i == threadCnt - 1)
- to = kafkaParts;
+ int counter = 0;
Review Comment:
Renamed the variable accroding to the rules
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 CDC: Improve Kafka partition ranges assignment for appliers [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1668658484
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -181,7 +172,7 @@ protected void runAppliers() {
stopped
);
- addAndStart("applier-thread-" + i, applier);
+ addAndStart("applier-thread-" + counter++, applier);
Review Comment:
Renamed the variable accroding to the rules
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 Algorithm improvement for kafka topic partition distribution [ignite-extensions]
Posted by "maksaska (via GitHub)" <gi...@apache.org>.
maksaska commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1666831448
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -197,6 +188,34 @@ protected void runAppliers() {
}
}
+ /**
+ * Creates Kafka partitions distribution for provided streamer configuration. Each element of the returning
+ * {@code List} is an {@link IgniteBiTuple}, which represents a kafka topic partition interval to be
+ * scanned by single thread.
+ * @param streamerCfg {@link KafkaToIgniteCdcStreamerConfiguration}.
+ * @return {@code List} of {@code IgniteBiTuple<Integer, Integer>}.
+ */
+ public static List<IgniteBiTuple<Integer, Integer>> getKafkaPartitionsDistribution(KafkaToIgniteCdcStreamerConfiguration streamerCfg) {
Review Comment:
Done!
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -197,6 +188,34 @@ protected void runAppliers() {
}
}
+ /**
+ * Creates Kafka partitions distribution for provided streamer configuration. Each element of the returning
+ * {@code List} is an {@link IgniteBiTuple}, which represents a kafka topic partition interval to be
+ * scanned by single thread.
+ * @param streamerCfg {@link KafkaToIgniteCdcStreamerConfiguration}.
+ * @return {@code List} of {@code IgniteBiTuple<Integer, Integer>}.
+ */
+ public static List<IgniteBiTuple<Integer, Integer>> getKafkaPartitionsDistribution(KafkaToIgniteCdcStreamerConfiguration streamerCfg) {
+ List<IgniteBiTuple<Integer, Integer>> distr = new ArrayList<>();
+
+ int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
+ int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
+ int threadCnt = streamerCfg.getThreadCount();
+
+ while (kafkaParts > 0) {
+ int partPerApplier = kafkaParts / threadCnt + (kafkaParts % threadCnt > 0 ? 1 : 0);
+
+ kafkaParts -= partPerApplier;
+ --threadCnt;
+
+ distr.add(new IgniteBiTuple<>(kafkaPartsFrom, kafkaPartsFrom + partPerApplier));
+
+ kafkaPartsFrom += partPerApplier;
+ }
+
+ return distr;
Review Comment:
Renamed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-22403 Algorithm improvement for kafka topic partition distribution [ignite-extensions]
Posted by "shishkovilja (via GitHub)" <gi...@apache.org>.
shishkovilja commented on code in PR #272:
URL: https://github.com/apache/ignite-extensions/pull/272#discussion_r1666807873
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -197,6 +188,34 @@ protected void runAppliers() {
}
}
+ /**
+ * Creates Kafka partitions distribution for provided streamer configuration. Each element of the returning
+ * {@code List} is an {@link IgniteBiTuple}, which represents a kafka topic partition interval to be
+ * scanned by single thread.
+ * @param streamerCfg {@link KafkaToIgniteCdcStreamerConfiguration}.
+ * @return {@code List} of {@code IgniteBiTuple<Integer, Integer>}.
+ */
+ public static List<IgniteBiTuple<Integer, Integer>> getKafkaPartitionsDistribution(KafkaToIgniteCdcStreamerConfiguration streamerCfg) {
+ List<IgniteBiTuple<Integer, Integer>> distr = new ArrayList<>();
Review Comment:
```suggestion
List<IgniteBiTuple<Integer, Integer>> parts = new ArrayList<>();
```
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -153,26 +154,16 @@ protected void runAppliers() {
streamerCfg
);
- int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
- int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
- int threadCnt = streamerCfg.getThreadCount();
-
- int partPerApplier = kafkaParts / threadCnt;
-
- for (int i = 0; i < threadCnt; i++) {
- int from = i * partPerApplier;
- int to = (i + 1) * partPerApplier;
-
- if (i == threadCnt - 1)
- to = kafkaParts;
+ int counter = 0;
+ for (IgniteBiTuple<Integer, Integer> part : getKafkaPartitionsDistribution(streamerCfg)) {
Review Comment:
```suggestion
for (IgniteBiTuple<Integer, Integer> parts : kafkaPartitions(streamerCfg)) {
```
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -153,26 +154,16 @@ protected void runAppliers() {
streamerCfg
);
- int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
- int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
- int threadCnt = streamerCfg.getThreadCount();
-
- int partPerApplier = kafkaParts / threadCnt;
-
- for (int i = 0; i < threadCnt; i++) {
- int from = i * partPerApplier;
- int to = (i + 1) * partPerApplier;
-
- if (i == threadCnt - 1)
- to = kafkaParts;
+ int counter = 0;
+ for (IgniteBiTuple<Integer, Integer> part : getKafkaPartitionsDistribution(streamerCfg)) {
KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
() -> eventsApplier(),
log,
kafkaProps,
streamerCfg.getTopic(),
- kafkaPartsFrom + from,
- kafkaPartsFrom + to,
+ part.get1(),
Review Comment:
```suggestion
parts.get1(), // kafkaPartFrom
```
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -197,6 +188,34 @@ protected void runAppliers() {
}
}
+ /**
+ * Creates Kafka partitions distribution for provided streamer configuration. Each element of the returning
+ * {@code List} is an {@link IgniteBiTuple}, which represents a kafka topic partition interval to be
+ * scanned by single thread.
+ * @param streamerCfg {@link KafkaToIgniteCdcStreamerConfiguration}.
+ * @return {@code List} of {@code IgniteBiTuple<Integer, Integer>}.
+ */
+ public static List<IgniteBiTuple<Integer, Integer>> getKafkaPartitionsDistribution(KafkaToIgniteCdcStreamerConfiguration streamerCfg) {
+ List<IgniteBiTuple<Integer, Integer>> distr = new ArrayList<>();
+
+ int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
+ int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
+ int threadCnt = streamerCfg.getThreadCount();
+
+ while (kafkaParts > 0) {
+ int partPerApplier = kafkaParts / threadCnt + (kafkaParts % threadCnt > 0 ? 1 : 0);
+
+ kafkaParts -= partPerApplier;
+ --threadCnt;
+
+ distr.add(new IgniteBiTuple<>(kafkaPartsFrom, kafkaPartsFrom + partPerApplier));
+
+ kafkaPartsFrom += partPerApplier;
+ }
+
+ return distr;
Review Comment:
```suggestion
return parts;
```
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -197,6 +188,34 @@ protected void runAppliers() {
}
}
+ /**
+ * Creates Kafka partitions distribution for provided streamer configuration. Each element of the returning
+ * {@code List} is an {@link IgniteBiTuple}, which represents a kafka topic partition interval to be
+ * scanned by single thread.
+ * @param streamerCfg {@link KafkaToIgniteCdcStreamerConfiguration}.
+ * @return {@code List} of {@code IgniteBiTuple<Integer, Integer>}.
+ */
+ public static List<IgniteBiTuple<Integer, Integer>> getKafkaPartitionsDistribution(KafkaToIgniteCdcStreamerConfiguration streamerCfg) {
Review Comment:
```suggestion
public static List<IgniteBiTuple<Integer, Integer>> kafkaPartitions(KafkaToIgniteCdcStreamerConfiguration streamerCfg) {
```
##########
modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java:
##########
@@ -153,26 +154,16 @@ protected void runAppliers() {
streamerCfg
);
- int kafkaPartsFrom = streamerCfg.getKafkaPartsFrom();
- int kafkaParts = streamerCfg.getKafkaPartsTo() - kafkaPartsFrom;
- int threadCnt = streamerCfg.getThreadCount();
-
- int partPerApplier = kafkaParts / threadCnt;
-
- for (int i = 0; i < threadCnt; i++) {
- int from = i * partPerApplier;
- int to = (i + 1) * partPerApplier;
-
- if (i == threadCnt - 1)
- to = kafkaParts;
+ int counter = 0;
+ for (IgniteBiTuple<Integer, Integer> part : getKafkaPartitionsDistribution(streamerCfg)) {
KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
() -> eventsApplier(),
log,
kafkaProps,
streamerCfg.getTopic(),
- kafkaPartsFrom + from,
- kafkaPartsFrom + to,
+ part.get1(),
+ part.get2(),
Review Comment:
```suggestion
parts.get2(), // kafkaPartTo
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org