You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2021/03/21 10:02:59 UTC
[GitHub] [ignite-extensions] nizhikov opened a new pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
nizhikov opened a new pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655277728
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Caches. */
+ private final Map<Integer, IgniteInternalCache<BinaryObject, BinaryObject>> ignCaches = new HashMap<>();
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Maximum batch size. */
+ private final int maxBatchSize;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
Review comment:
It's a common way to deal with boundaries.
Please, take a look at other standard JDK methods:
```
/**
beginIndex – the beginning index, inclusive.
endIndex – the ending index, exclusive.
*/
String#substring(int, int);
/**
beginIndex – the begin index, inclusive.
endIndex – the end index, exclusive.
*/
String#subSequence(int, int);
/**
fromIndex – low endpoint (inclusive) of the subList
toIndex – high endpoint (exclusive) of the subList
*/
List#subList(int, int);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652488371
##########
File path: modules/cdc-ext/modules/core/src/test/config/tests.properties
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
Review comment:
That still not an answer why, for example, we have to define
`glassfish.jmx.rmi.connector.port=8686` or `hadoop.home=@{HADOOP_HOME}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652443729
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
Review comment:
Could names be more understandable?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ o = ((BinaryObject)oldVal).field(conflictResolveField);
+ n = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ o = U.field(oldVal, conflictResolveField);
+ n = U.field(newVal, conflictResolveField);
+ }
+
+ if (o == null || n == null)
+ return o == null;
+
+ return o.compareTo(n) < 0;
+ }
+ catch (Exception e) {
+ log.error("Error while resolving replication conflict with field '" +
+ conflictResolveField + "'.\nConflict resolve with the field disabled.", e);
+
+ conflictResolveFieldEnabled = false;
+ }
+ }
+ }
+
+ // Cluster with the lower ID have biggest priority(e.g. first cluster is main).
+ return newEntry.dataCenterId() < oldEntry.dataCenterId();
Review comment:
> Conflict resolver is extremely hot code so I don't think we can insert logging here.
This is the rarest case when hot cases skipped.
> Yes, please, take a look at IgniteSystemProperties#IGNITE_TO_STRING_INCLUDE_SENSITIVE
This flag works only when you use it %)
For example you forget to use it at CacheVersionConflictResolverImpl#resolve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655531203
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.CdcEventsApplier;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier extends CdcEventsApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
+
+ /** Caches ids to read. */
+ private final Set<Integer> caches;
+
+ /** Consumers. */
+ private final List<KafkaConsumer<Integer, byte[]>> cnsmrs = new ArrayList<>();
+
+ /** */
+ private final AtomicLong rcvdEvts = new AtomicLong();
+
+ /**
+ * @param ign Ignite instance.
+ * @param log Logger.
+ * @param kafkaProps Kafka properties.
+ * @param topic Topic name.
+ * @param kafkaPartFrom Read from partition.
+ * @param kafkaPartTo Read to partition.
+ * @param caches Cache ids.
+ * @param maxBatchSize Maximum batch size.
+ * @param closed Closed flag.
+ */
+ public KafkaToIgniteCdcStreamerApplier(
+ IgniteEx ign,
+ IgniteLogger log,
+ Properties kafkaProps,
+ String topic,
+ int kafkaPartFrom,
+ int kafkaPartTo,
+ Set<Integer> caches,
+ int maxBatchSize,
+ AtomicBoolean closed
+ ) {
+ super(maxBatchSize);
+
+ this.ign = ign;
+ this.kafkaProps = kafkaProps;
+ this.topic = topic;
+ this.kafkaPartFrom = kafkaPartFrom;
+ this.kafkaPartTo = kafkaPartTo;
+ this.caches = caches;
+ this.closed = closed;
+ this.log = log.getLogger(KafkaToIgniteCdcStreamerApplier.class);
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ U.setCurrentIgniteName(ign.name());
+
+ try {
+ for (int kafkaPart = kafkaPartFrom; kafkaPart < kafkaPartTo; kafkaPart++) {
+ KafkaConsumer<Integer, byte[]> cnsmr = new KafkaConsumer<>(kafkaProps);
+
+ cnsmr.assign(Collections.singleton(new TopicPartition(topic, kafkaPart)));
+
+ cnsmrs.add(cnsmr);
+ }
+
+ Iterator<KafkaConsumer<Integer, byte[]>> cnsmrIter = Collections.emptyIterator();
+
+ while (!closed.get()) {
+ if (!cnsmrIter.hasNext())
+ cnsmrIter = cnsmrs.iterator();
+
+ poll(cnsmrIter.next());
+ }
+ }
+ catch (WakeupException e) {
+ if (!closed.get())
+ log.error("Applier wakeup error!", e);
+ }
+ catch (Throwable e) {
+ log.error("Applier error!", e);
+
+ closed.set(true);
Review comment:
I don't think so. Why?
All other appliers will try to perform last poll and will finish after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652424609
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ o = ((BinaryObject)oldVal).field(conflictResolveField);
+ n = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ o = U.field(oldVal, conflictResolveField);
+ n = U.field(newVal, conflictResolveField);
+ }
+
+ if (o == null || n == null)
Review comment:
Conflict resolver is extremely hot code so I don't think we can insert logging here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656148235
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = new IgniteEx[] {
+ startGrid(1),
+ startGrid(2)
+ };
+
+ srcClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ srcClusterCliCfg[i] = optimize(getConfiguration("src-cluster-client" + i).setClientMode(true));
+
+ srcCluster[0].cluster().state(ACTIVE);
+ srcCluster[0].cluster().tag("source");
Review comment:
should "source" be const?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
Review comment:
method content describes 2 pretty similar operations (cluster configurations).
could we deduplicate this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648059838
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
Review comment:
Just ask yourself what is ChangeDataCaptureIgniteToKafka
What role it has, what it do?
The name should explain this.
It's a consumer because inherited from a consumer.
It's a KafkaConsumer because it's consuming to Kafka.
It's already clear that it consumes from Ignite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652439261
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.checkCRC;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.generateSingleData;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER},
+ {TRANSACTIONAL, THIRD_CLUSTER},
+ {ATOMIC, FIRST_CLUSTER},
+ {TRANSACTIONAL, FIRST_CLUSTER},
Review comment:
Of-course it will
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652423583
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.checkCRC;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.generateSingleData;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER},
+ {TRANSACTIONAL, THIRD_CLUSTER},
+ {ATOMIC, FIRST_CLUSTER},
+ {TRANSACTIONAL, FIRST_CLUSTER},
Review comment:
This change doesn't make code simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652443070
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ o = ((BinaryObject)oldVal).field(conflictResolveField);
+ n = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ o = U.field(oldVal, conflictResolveField);
+ n = U.field(newVal, conflictResolveField);
+ }
+
+ if (o == null || n == null)
Review comment:
Sound like Apache Ignite is extremely fast, we have no time to handle exceptions :)
Here you comparing by field but one or both objects do not have it.
And that's a contract violation, you may even throw an exception here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656161873
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = new IgniteEx[] {
+ startGrid(1),
+ startGrid(2)
+ };
+
+ srcClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ srcClusterCliCfg[i] = optimize(getConfiguration("src-cluster-client" + i).setClientMode(true));
+
+ srcCluster[0].cluster().state(ACTIVE);
+ srcCluster[0].cluster().tag("source");
Review comment:
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652452861
##########
File path: modules/cdc-ext/modules/core/src/test/config/tests.properties
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
Review comment:
Can't do it - `GridCommonAbstractTest` will not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652521967
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable oldResolveField;
+ Comparable newResolveField;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ oldResolveField = ((BinaryObject)oldVal).field(conflictResolveField);
+ newResolveField = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ oldResolveField = U.field(oldVal, conflictResolveField);
+ newResolveField = U.field(newVal, conflictResolveField);
+ }
+
+ if (oldResolveField == null || newResolveField == null)
+ return oldResolveField == null;
+
+ return oldResolveField.compareTo(newResolveField) < 0;
+ }
+ catch (Exception e) {
Review comment:
There may be an exception in user-provided Comparable implementation.
I don't think Ignite cluster should fail in case of not correct user code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656187555
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/CdcKafkaReplicationTest.java
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import org.apache.ignite.cdc.AbstractReplicationTest;
+import org.apache.ignite.cdc.ChangeDataCaptureConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.junit.ClassRule;
+
+import static org.apache.ignite.cdc.kafka.KafkaToIgniteCdcStreamerConfiguration.DFLT_PARTS;
+import static org.apache.ignite.cdc.kafka.KafkaToIgniteCdcStreamerConfiguration.DFLT_TOPIC;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+
+/**
+ * Tests for kafka replication.
+ */
+public class CdcKafkaReplicationTest extends AbstractReplicationTest {
+ /** */
+ public static final String SRC_DEST_TOPIC = "source-dest";
+
+ /** */
+ public static final String DEST_SRC_TOPIC = "dest-source";
+
+ /** */
+ protected static Properties props;
+
+ /** */
+ @ClassRule
+ public static final EmbeddedKafkaCluster KAFKA = new EmbeddedKafkaCluster(1);
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ clientsCnt = 2;
+
+ super.beforeTest();
+
+ KAFKA.start();
+
+ if (props == null) {
+ props = new Properties();
+
+ props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA.bootstrapServers());
+ props.put(ConsumerConfig.GROUP_ID_CONFIG, "kafka-to-ignite-applier");
+ props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
+ props.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, "10000");
+ }
+
+ KAFKA.createTopic(DFLT_TOPIC, DFLT_PARTS, 1);
+ KAFKA.createTopic(SRC_DEST_TOPIC, DFLT_PARTS, 1);
+ KAFKA.createTopic(DEST_SRC_TOPIC, DFLT_PARTS, 1);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ props = null;
+
+ KAFKA.deleteAllTopicsAndWait(getTestTimeout());
+ }
+
+ /** {@inheritDoc} */
+ @Override protected List<IgniteInternalFuture<?>> startActivePassiveCdc() {
+ List<IgniteInternalFuture<?>> futs = new ArrayList<>();
+
+ futs.add(igniteToKafka(srcCluster[0].configuration(), DFLT_TOPIC, AbstractReplicationTest.AP_CACHE));
+ futs.add(igniteToKafka(srcCluster[1].configuration(), DFLT_TOPIC, AbstractReplicationTest.AP_CACHE));
Review comment:
should be replaced with loop iteration on `clientsCnt`
as well as other duplicated calls
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = new IgniteEx[] {
+ startGrid(1),
+ startGrid(2)
+ };
+
+ srcClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ srcClusterCliCfg[i] = optimize(getConfiguration("src-cluster-client" + i).setClientMode(true));
+
+ srcCluster[0].cluster().state(ACTIVE);
+ srcCluster[0].cluster().tag("source");
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ commPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = new IgniteEx[] {
+ startGrid(4),
+ startGrid(5)
+ };
+
+ destClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ destClusterCliCfg[i] = optimize(getConfiguration("dest-cluster-client" + i).setClientMode(true));
+
+ assertFalse("source".equals(destCluster[0].cluster().tag()));
+
+ destCluster[0].cluster().state(ACTIVE);
+ destCluster[0].cluster().tag("destination");
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ List<IgniteInternalFuture<?>> futs = startActivePassiveCdc();
+
+ try {
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster[0].createCache(AP_CACHE);
+
+ destCache.put(1, ConflictResolvableTestData.create());
+ destCache.remove(1);
+
+ // Updates for "ignored-cache" should be ignored because of CDC consume configuration.
+ runAsync(generateData("ignored-cache", srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT)));
+ runAsync(generateData(AP_CACHE, srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT)));
+
+ List<IgniteInternalFuture<?>> k2iFut = startActivePassiveReplication();
+
+ if (k2iFut != null)
+ futs.addAll(k2iFut);
+
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster[srcCluster.length - 1].getOrCreateCache(AP_CACHE);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, EXISTS, futs);
+
+ IntStream.range(0, KEYS_CNT).forEach(srcCache::remove);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ @Test
+ public void testActiveActiveReplication() throws Exception {
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, srcCluster[srcCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 == 0)));
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, destCluster[destCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 != 0)));
+
+ List<IgniteInternalFuture<?>> futs = startActiveActiveCdc();
+
+ try {
+ List<IgniteInternalFuture<?>> replicationFuts = startActiveActiveReplication();
+
+ if (replicationFuts != null)
+ futs.addAll(replicationFuts);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, EXISTS, futs);
+
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 == 0).forEach(srcCache::remove));
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 != 0).forEach(destCache::remove));
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ public static Runnable generateData(String cacheName, IgniteEx ign, IntStream keys) {
+ return () -> {
+ IgniteCache<Integer, ConflictResolvableTestData> cache = ign.getOrCreateCache(cacheName);
+
+ keys.forEach(i -> cache.put(i, ConflictResolvableTestData.create()));
+ };
+ }
+
+ /** */
+ public void waitForSameData(
+ IgniteCache<Integer, ConflictResolvableTestData> src,
+ IgniteCache<Integer, ConflictResolvableTestData> dest,
+ int keysCnt,
+ int keysState,
+ List<IgniteInternalFuture<?>> futs
+ ) throws IgniteInterruptedCheckedException {
+ assertTrue(waitForCondition(() -> {
+ for (int i = 0; i < keysCnt; i++) {
+ if (keysState == EXISTS) {
+ if (!src.containsKey(i) || !dest.containsKey(i))
+ return checkFuts(false, futs);
+ }
+ else if (keysState == REMOVED) {
+ if (src.containsKey(i) || dest.containsKey(i))
+ return checkFuts(false, futs);
+
+ continue;
+ }
+ else
+ throw new IllegalArgumentException(keysState + " not supported.");
+
+ ConflictResolvableTestData data = dest.get(i);
+
+ if (!data.equals(src.get(i)))
+ return checkFuts(false, futs);
+ }
+
+ return checkFuts(true, futs);
+ }, getTestTimeout()));
+ }
+
+ /** */
+ private boolean checkFuts(boolean res, List<IgniteInternalFuture<?>> futs) {
+ for (int i = 0; i < futs.size(); i++)
+ assertFalse("Fut " + i, futs.get(i).isDone());
+
+ for (IgniteInternalFuture<?> fut : futs)
+ assertFalse(fut.isDone());
+
+ return res;
+ }
+
+ /** */
+ protected abstract List<IgniteInternalFuture<?>> startActivePassiveCdc();
+
+ /** */
+ protected abstract List<IgniteInternalFuture<?>> startActiveActiveCdc();
Review comment:
TwoWay?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CdcIgniteToIgniteReplicationTest.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+
+/** */
+public class CdcIgniteToIgniteReplicationTest extends AbstractReplicationTest {
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ clientsCnt = 2;
+
+ super.beforeTest();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected List<IgniteInternalFuture<?>> startActivePassiveCdc() {
+ List<IgniteInternalFuture<?>> futs = new ArrayList<>();
+
+ futs.add(igniteToIgnite(srcCluster[0].configuration(), destClusterCliCfg[0], AP_CACHE));
+ futs.add(igniteToIgnite(srcCluster[1].configuration(), destClusterCliCfg[1], AP_CACHE));
Review comment:
could this be simplified with a loop over `clientsCnt` incapsulated at method with 3 usages at this class?
one here, at 2 at `startActiveActiveCdc()`
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = new IgniteEx[] {
+ startGrid(1),
+ startGrid(2)
+ };
+
+ srcClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ srcClusterCliCfg[i] = optimize(getConfiguration("src-cluster-client" + i).setClientMode(true));
+
+ srcCluster[0].cluster().state(ACTIVE);
+ srcCluster[0].cluster().tag("source");
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ commPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = new IgniteEx[] {
+ startGrid(4),
+ startGrid(5)
+ };
+
+ destClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ destClusterCliCfg[i] = optimize(getConfiguration("dest-cluster-client" + i).setClientMode(true));
+
+ assertFalse("source".equals(destCluster[0].cluster().tag()));
+
+ destCluster[0].cluster().state(ACTIVE);
+ destCluster[0].cluster().tag("destination");
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ List<IgniteInternalFuture<?>> futs = startActivePassiveCdc();
+
+ try {
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster[0].createCache(AP_CACHE);
+
+ destCache.put(1, ConflictResolvableTestData.create());
+ destCache.remove(1);
+
+ // Updates for "ignored-cache" should be ignored because of CDC consume configuration.
+ runAsync(generateData("ignored-cache", srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT)));
+ runAsync(generateData(AP_CACHE, srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT)));
+
+ List<IgniteInternalFuture<?>> k2iFut = startActivePassiveReplication();
+
+ if (k2iFut != null)
+ futs.addAll(k2iFut);
+
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster[srcCluster.length - 1].getOrCreateCache(AP_CACHE);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, EXISTS, futs);
+
+ IntStream.range(0, KEYS_CNT).forEach(srcCache::remove);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ @Test
+ public void testActiveActiveReplication() throws Exception {
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, srcCluster[srcCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 == 0)));
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, destCluster[destCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 != 0)));
+
+ List<IgniteInternalFuture<?>> futs = startActiveActiveCdc();
+
+ try {
+ List<IgniteInternalFuture<?>> replicationFuts = startActiveActiveReplication();
+
+ if (replicationFuts != null)
+ futs.addAll(replicationFuts);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, EXISTS, futs);
+
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 == 0).forEach(srcCache::remove));
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 != 0).forEach(destCache::remove));
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ public static Runnable generateData(String cacheName, IgniteEx ign, IntStream keys) {
+ return () -> {
+ IgniteCache<Integer, ConflictResolvableTestData> cache = ign.getOrCreateCache(cacheName);
+
+ keys.forEach(i -> cache.put(i, ConflictResolvableTestData.create()));
+ };
+ }
+
+ /** */
+ public void waitForSameData(
+ IgniteCache<Integer, ConflictResolvableTestData> src,
+ IgniteCache<Integer, ConflictResolvableTestData> dest,
+ int keysCnt,
+ int keysState,
+ List<IgniteInternalFuture<?>> futs
+ ) throws IgniteInterruptedCheckedException {
+ assertTrue(waitForCondition(() -> {
+ for (int i = 0; i < keysCnt; i++) {
+ if (keysState == EXISTS) {
+ if (!src.containsKey(i) || !dest.containsKey(i))
+ return checkFuts(false, futs);
+ }
+ else if (keysState == REMOVED) {
+ if (src.containsKey(i) || dest.containsKey(i))
+ return checkFuts(false, futs);
+
+ continue;
+ }
+ else
+ throw new IllegalArgumentException(keysState + " not supported.");
+
+ ConflictResolvableTestData data = dest.get(i);
+
+ if (!data.equals(src.get(i)))
+ return checkFuts(false, futs);
+ }
+
+ return checkFuts(true, futs);
+ }, getTestTimeout()));
+ }
+
+ /** */
+ private boolean checkFuts(boolean res, List<IgniteInternalFuture<?>> futs) {
+ for (int i = 0; i < futs.size(); i++)
+ assertFalse("Fut " + i, futs.get(i).isDone());
+
+ for (IgniteInternalFuture<?> fut : futs)
+ assertFalse(fut.isDone());
+
+ return res;
+ }
+
+ /** */
+ protected abstract List<IgniteInternalFuture<?>> startActivePassiveCdc();
Review comment:
OneWay?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String ACTIVE_PASSIVE_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 1;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 2;
+
+ /** */
+ private enum WaitDataMode {
+ /** */
+ EXISTS,
+
+ /** */
+ REMOVED
+ }
+
+ /** */
+ public static final int KEYS_CNT = 1000;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> srcCluster;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> destCluster;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = setupCluster("source", "src-cluster-client", 0);
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ commPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = setupCluster("destination", "dest-cluster-client", 2);
+ }
+
+ /** */
+ private IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> setupCluster(String clusterTag, String clientPrefix, int idx) throws Exception {
+ IgniteEx[] cluster = new IgniteEx[] {
+ startGrid(idx + 1),
+ startGrid(idx + 2)
+ };
+
+ IgniteConfiguration[] clusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < 2; i++)
+ clusterCliCfg[i] = optimize(getConfiguration(clientPrefix + i).setClientMode(true));
+
+ cluster[0].cluster().state(ACTIVE);
+ cluster[0].cluster().tag(clusterTag);
+
+ return F.t(cluster, clusterCliCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** Active/Passive mode means changes made only in one cluster. */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ List<IgniteInternalFuture<?>> futs = startActivePassiveCdc();
+
+ try {
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster.get1()[0].createCache(ACTIVE_PASSIVE_CACHE);
+
+ destCache.put(1, ConflictResolvableTestData.create());
+ destCache.remove(1);
+
+ // Updates for "ignored-cache" should be ignored because of CDC consume configuration.
+ runAsync(generateData("ignored-cache", srcCluster.get1()[srcCluster.get1().length - 1], IntStream.range(0, KEYS_CNT)));
+ runAsync(generateData(ACTIVE_PASSIVE_CACHE, srcCluster.get1()[srcCluster.get1().length - 1], IntStream.range(0, KEYS_CNT)));
+
+ List<IgniteInternalFuture<?>> k2iFut = startActivePassiveReplication();
+
+ if (k2iFut != null)
+ futs.addAll(k2iFut);
+
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache =
+ srcCluster.get1()[srcCluster.get1().length - 1].getOrCreateCache(ACTIVE_PASSIVE_CACHE);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, WaitDataMode.EXISTS, futs);
+
+ IntStream.range(0, KEYS_CNT).forEach(srcCache::remove);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, WaitDataMode.REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** Active/Active mode means changes made in both clusters. */
+ @Test
+ public void testActiveActiveReplication() throws Exception {
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster.get1()[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster.get1()[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, srcCluster.get1()[srcCluster.get1().length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 == 0)));
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, destCluster.get1()[destCluster.get1().length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 != 0)));
+
+ List<IgniteInternalFuture<?>> futs = startActiveActiveCdc();
+
+ try {
+ List<IgniteInternalFuture<?>> replicationFuts = startActiveActiveReplication();
+
+ if (replicationFuts != null)
+ futs.addAll(replicationFuts);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, WaitDataMode.EXISTS, futs);
+
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 == 0).forEach(srcCache::remove));
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 != 0).forEach(destCache::remove));
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, WaitDataMode.REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ public static Runnable generateData(String cacheName, IgniteEx ign, IntStream keys) {
+ return () -> {
+ IgniteCache<Integer, ConflictResolvableTestData> cache = ign.getOrCreateCache(cacheName);
+
+ keys.forEach(i -> cache.put(i, ConflictResolvableTestData.create()));
+ };
+ }
+
+ /** */
+ public void waitForSameData(
+ IgniteCache<Integer, ConflictResolvableTestData> src,
+ IgniteCache<Integer, ConflictResolvableTestData> dest,
+ int keysCnt,
+ WaitDataMode mode,
+ List<IgniteInternalFuture<?>> futs
+ ) throws IgniteInterruptedCheckedException {
+ assertTrue(waitForCondition(() -> {
+ for (int i = 0; i < keysCnt; i++) {
+ if (mode == WaitDataMode.EXISTS) {
+ if (!src.containsKey(i) || !dest.containsKey(i))
+ return checkFuts(false, futs);
+ }
+ else if (mode == WaitDataMode.REMOVED) {
+ if (src.containsKey(i) || dest.containsKey(i))
+ return checkFuts(false, futs);
+
+ continue;
+ }
+ else
+ throw new IllegalArgumentException(mode + " not supported.");
+
+ ConflictResolvableTestData data = dest.get(i);
+
+ if (!data.equals(src.get(i)))
+ return checkFuts(false, futs);
+ }
+
+ return checkFuts(true, futs);
+ }, getTestTimeout()));
+ }
+
+ /** */
+ private boolean checkFuts(boolean res, List<IgniteInternalFuture<?>> futs) {
+ for (int i = 0; i < futs.size(); i++)
+ assertFalse("Fut " + i, futs.get(i).isDone());
+
+ for (IgniteInternalFuture<?> fut : futs)
+ assertFalse(fut.isDone());
+
+ return res;
+ }
+
+ /** */
+ protected abstract List<IgniteInternalFuture<?>> startActivePassiveCdc();
+
+ /** */
+ protected abstract List<IgniteInternalFuture<?>> startActiveActiveCdc();
+
+ /** */
+ protected List<IgniteInternalFuture<?>> startActivePassiveReplication() {
+ return null;
+ }
+
+ /** */
+ protected List<IgniteInternalFuture<?>> startActiveActiveReplication() {
+ return null;
+ }
+}
Review comment:
looks like these methods should be the part of `start*Cdc()` methods above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652452012
##########
File path: modules/cdc-ext/modules/core/src/test/config/log4j-test.xml
##########
@@ -0,0 +1,145 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE log4j:configuration PUBLIC "-//APACHE//DTD LOG4J 1.2//EN"
Review comment:
Please do this at your PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655277728
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Caches. */
+ private final Map<Integer, IgniteInternalCache<BinaryObject, BinaryObject>> ignCaches = new HashMap<>();
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Maximum batch size. */
+ private final int maxBatchSize;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
Review comment:
It's a common way to deal with bounaries.
Please, take a look at other standard JDK methods:
```
/**
beginIndex – the beginning index, inclusive.
endIndex – the ending index, exclusive.
*/
String#substring(int, int);
/**
beginIndex – the begin index, inclusive.
endIndex – the end index, exclusive.
*/
String#subSequence(int, int);
/**
fromIndex – low endpoint (inclusive) of the subList
toIndex – high endpoint (exclusive) of the subList
*/
List#subList(int, int);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r647416657
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/KafkaUtils.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.DescribeTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.common.errors.TopicExistsException;
+import org.apache.kafka.common.errors.UnknownTopicOrPartitionException;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/** Kafka Utils. */
+public class KafkaUtils {
+ /** Default kafka request timeout. */
+ public static final int TIMEOUT_MIN = 1;
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_NUM_PARTITIONS = "ignite.to.kafka.numpartitions";
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_REPLICATION_FACTOR = "ignite.to.kafka.replication.factor";
Review comment:
The proper place for this property is a `ChangeDataCaptureIgniteToKafka`
and this means you can get rid of IGNITE_TO_KAFKA` prefix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648062199
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureKafkaToIgnite.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.cdc.conflictplugin.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link ChangeDataCaptureIgniteToKafka} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * Properties list:
+ * <ul>
+ * <li>{@link #KAFKA_TO_IGNITE_THREAD_COUNT} - count of {@link Applier} threads.</li>
+ * <li>{@link #IGNITE_TO_KAFKA_TOPIC} - Kafka topic name if not provided in constructor.</li>
+ * </ul>
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class ChangeDataCaptureKafkaToIgnite implements Runnable {
Review comment:
> What role it has, what it do?
Consumes CDC events from Kafka topic and applies it to Ignite cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r654342271
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Conflict can't be resolved. Update ignored. Old version used.</li>
+ * </ul>
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else
+ res.useOld();
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable oldResolveField;
+ Comparable newResolveField;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ oldResolveField = ((BinaryObject)oldVal).field(conflictResolveField);
+ newResolveField = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ oldResolveField = U.field(oldVal, conflictResolveField);
+ newResolveField = U.field(newVal, conflictResolveField);
+ }
+
+ return oldResolveField.compareTo(newResolveField) < 0;
+ }
+ catch (Exception e) {
+ log.error(
+ "Error while resolving replication conflict. [field=" + conflictResolveField + ", key=" + newEntry.key() + ']',
+ e
+ );
+ }
+ }
+ }
+
+ log.error("Conflict can't be resolved, update ignored [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ", conflictResolveField=" + conflictResolveField + ']');
Review comment:
conflictResolveField is absent here?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Conflict can't be resolved. Update ignored. Old version used.</li>
+ * </ul>
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else
+ res.useOld();
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable oldResolveField;
+ Comparable newResolveField;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ oldResolveField = ((BinaryObject)oldVal).field(conflictResolveField);
+ newResolveField = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ oldResolveField = U.field(oldVal, conflictResolveField);
+ newResolveField = U.field(newVal, conflictResolveField);
+ }
+
+ return oldResolveField.compareTo(newResolveField) < 0;
+ }
+ catch (Exception e) {
+ log.error(
+ "Error while resolving replication conflict. [field=" + conflictResolveField + ", key=" + newEntry.key() + ']',
+ e
+ );
+ }
+ }
+ }
+
+ log.error("Conflict can't be resolved, update ignored [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
Review comment:
Some hints of how to fix this issue should be provided.
1) Fix this by manual update
or
2) Use conflictResolveField
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheEntry;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Other cluster id. */
+ @Parameterized.Parameter(1)
+ public byte otherClusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, otherClusterId={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (byte otherClusterId : new byte[] {FIRST_CLUSTER_ID, THIRD_CLUSTER_ID})
+ params.add(new Object[] {mode, otherClusterId});
+
+ return params;
+ }
+
+ /** */
+ private static IgniteCache<String, ConflictResolvableTestData> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName).setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, ConflictResolvableTestData>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = key("UpdateFromOtherClusterWithoutConflict", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ putConflict(key, 2, true);
+
+ removeConflict(key, 3, true);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = key("UpdateClusterUpdateReorder", otherClusterId);
+
+ putConflict(key, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putConflict(key, 2, false);
+ putConflict(key, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removeConflict(key, 2, false);
+ removeConflict(key, 1, false);
+
+ // Remove with the higher order should succeed.
+ putConflict(key, 3, true);
+
+ key = key("UpdateClusterUpdateReorder2", otherClusterId);
+
+ int order = 1;
+
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), true);
+
+ // Update with the equal or lower topVer should fail.
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower topVer should fail.
+ removeConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the higher topVer should succeed.
+ putConflict(key, new GridCacheVersion(3, order, 1, otherClusterId), true);
+
+ key = key("UpdateClusterUpdateReorder3", otherClusterId);
+
+ int topVer = 1;
+
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), true);
+
+ // Update with the equal or lower nodeOrder should fail.
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower nodeOrder should fail.
+ removeConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the higher nodeOrder should succeed.
+ putConflict(key, new GridCacheVersion(topVer, order, 3, otherClusterId), true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = key("UpdateThisClusterConflict0", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key);
+
+ // Conflict replicated update should fail.
+ putConflict(key, 2, false);
+
+ key = key("UpdateThisDCConflict1", otherClusterId);
+
+ putConflict(key, 3, true);
+
+ // Local update for other cluster entry should succeed.
+ put(key);
+
+ key = key("UpdateThisDCConflict2", otherClusterId);
+
+ put(key);
+
+ // Conflict replicated remove should fail.
+ removeConflict(key, 4, false);
+
+ key = key("UpdateThisDCConflict3", otherClusterId);
+
+ put(key);
+
+ // Conflict replicated update succeed only if resolved by field.
+ putConflict(key, 5, conflictResolveField() != null);
+ }
+
+ /** */
+ private void put(String key) {
+ ConflictResolvableTestData newVal = ConflictResolvableTestData.create();
+
+ CacheEntry<String, ConflictResolvableTestData> oldEntry = cache.getEntry(key);
+
+ cache.put(key, newVal);
+
+ CacheEntry<String, ConflictResolvableTestData> newEntry = cache.getEntry(key);
+
+ assertNull(((CacheEntryVersion)newEntry.version()).otherClusterVersion());
+ assertEquals(newVal, cache.get(key));
+
+ if (oldEntry != null)
+ assertTrue(((CacheEntryVersion)oldEntry.version()).order() < ((CacheEntryVersion)newEntry.version()).order());
+ }
+
+ /** Puts entry via {@link IgniteInternalCache#putAllConflict(Map)}. */
+ private void putConflict(String k, long order, boolean success) throws IgniteCheckedException {
+ putConflict(k, new GridCacheVersion(1, order, 1, otherClusterId), success);
+ }
+
+ /** Puts entry via {@link IgniteInternalCache#putAllConflict(Map)}. */
+ private void putConflict(String k, GridCacheVersion newVer, boolean success) throws IgniteCheckedException {
+ CacheEntry<String, ConflictResolvableTestData> oldVal = cache.getEntry(k);
+ ConflictResolvableTestData newVal = ConflictResolvableTestData.create();
+
+ KeyCacheObject key = new KeyCacheObjectImpl(k, null, cachex.context().affinity().partition(k));
+ CacheObject val = new CacheObjectImpl(client.binary().toBinary(newVal), null);
+
+ cachex.putAllConflict(singletonMap(key, new GridCacheDrInfo(val, newVer)));
+
+ if (success) {
+ assertEquals(newVer, ((CacheEntryVersion)cache.getEntry(k).version()).otherClusterVersion());
+ assertEquals(newVal, cache.get(k));
+ } else if (oldVal != null) {
+ assertEquals(oldVal.getValue(), cache.get(k));
+ assertEquals(oldVal.version(), cache.getEntry(k).version());
+ }
+ }
+
+ /** */
+ private void remove(String key) {
+ cache.remove(key);
+
+ assertFalse(cache.containsKey(key));
+ }
+
+ /** Removes entry via {@link IgniteInternalCache#removeAllConflict(Map)}. */
+ private void removeConflict(String k, long order, boolean success) throws IgniteCheckedException {
+ removeConflict(k, new GridCacheVersion(1, order, 1, otherClusterId), success);
+ }
+
+ /** Removes entry via {@link IgniteInternalCache#removeAllConflict(Map)}. */
+ private void removeConflict(String k, GridCacheVersion ver, boolean success) throws IgniteCheckedException {
Review comment:
assertion we have the value before the remove will be useful.
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheEntry;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Other cluster id. */
+ @Parameterized.Parameter(1)
+ public byte otherClusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, otherClusterId={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (byte otherClusterId : new byte[] {FIRST_CLUSTER_ID, THIRD_CLUSTER_ID})
+ params.add(new Object[] {mode, otherClusterId});
+
+ return params;
+ }
+
+ /** */
+ private static IgniteCache<String, ConflictResolvableTestData> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName).setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, ConflictResolvableTestData>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = key("UpdateFromOtherClusterWithoutConflict", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ putConflict(key, 2, true);
+
+ removeConflict(key, 3, true);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = key("UpdateClusterUpdateReorder", otherClusterId);
+
+ putConflict(key, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putConflict(key, 2, false);
+ putConflict(key, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removeConflict(key, 2, false);
+ removeConflict(key, 1, false);
+
+ // Remove with the higher order should succeed.
+ putConflict(key, 3, true);
+
+ key = key("UpdateClusterUpdateReorder2", otherClusterId);
+
+ int order = 1;
+
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), true);
+
+ // Update with the equal or lower topVer should fail.
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower topVer should fail.
+ removeConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the higher topVer should succeed.
+ putConflict(key, new GridCacheVersion(3, order, 1, otherClusterId), true);
+
+ key = key("UpdateClusterUpdateReorder3", otherClusterId);
+
+ int topVer = 1;
+
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), true);
+
+ // Update with the equal or lower nodeOrder should fail.
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower nodeOrder should fail.
+ removeConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the higher nodeOrder should succeed.
+ putConflict(key, new GridCacheVersion(topVer, order, 3, otherClusterId), true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = key("UpdateThisClusterConflict0", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key);
+
+ // Conflict replicated update should fail.
+ putConflict(key, 2, false);
+
+ key = key("UpdateThisDCConflict1", otherClusterId);
+
+ putConflict(key, 3, true);
+
+ // Local update for other cluster entry should succeed.
+ put(key);
+
+ key = key("UpdateThisDCConflict2", otherClusterId);
+
+ put(key);
+
+ // Conflict replicated remove should fail.
+ removeConflict(key, 4, false);
+
+ key = key("UpdateThisDCConflict3", otherClusterId);
+
+ put(key);
+
+ // Conflict replicated update succeed only if resolved by field.
+ putConflict(key, 5, conflictResolveField() != null);
+ }
+
+ /** */
+ private void put(String key) {
+ ConflictResolvableTestData newVal = ConflictResolvableTestData.create();
+
+ CacheEntry<String, ConflictResolvableTestData> oldEntry = cache.getEntry(key);
+
+ cache.put(key, newVal);
+
+ CacheEntry<String, ConflictResolvableTestData> newEntry = cache.getEntry(key);
+
+ assertNull(((CacheEntryVersion)newEntry.version()).otherClusterVersion());
+ assertEquals(newVal, cache.get(key));
+
+ if (oldEntry != null)
+ assertTrue(((CacheEntryVersion)oldEntry.version()).order() < ((CacheEntryVersion)newEntry.version()).order());
+ }
+
+ /** Puts entry via {@link IgniteInternalCache#putAllConflict(Map)}. */
+ private void putConflict(String k, long order, boolean success) throws IgniteCheckedException {
+ putConflict(k, new GridCacheVersion(1, order, 1, otherClusterId), success);
+ }
+
+ /** Puts entry via {@link IgniteInternalCache#putAllConflict(Map)}. */
+ private void putConflict(String k, GridCacheVersion newVer, boolean success) throws IgniteCheckedException {
+ CacheEntry<String, ConflictResolvableTestData> oldVal = cache.getEntry(k);
+ ConflictResolvableTestData newVal = ConflictResolvableTestData.create();
+
+ KeyCacheObject key = new KeyCacheObjectImpl(k, null, cachex.context().affinity().partition(k));
+ CacheObject val = new CacheObjectImpl(client.binary().toBinary(newVal), null);
+
+ cachex.putAllConflict(singletonMap(key, new GridCacheDrInfo(val, newVer)));
+
+ if (success) {
+ assertEquals(newVer, ((CacheEntryVersion)cache.getEntry(k).version()).otherClusterVersion());
+ assertEquals(newVal, cache.get(k));
+ } else if (oldVal != null) {
+ assertEquals(oldVal.getValue(), cache.get(k));
+ assertEquals(oldVal.version(), cache.getEntry(k).version());
+ }
+ }
+
+ /** */
+ private void remove(String key) {
+ cache.remove(key);
+
+ assertFalse(cache.containsKey(key));
+ }
+
+ /** Removes entry via {@link IgniteInternalCache#removeAllConflict(Map)}. */
+ private void removeConflict(String k, long order, boolean success) throws IgniteCheckedException {
+ removeConflict(k, new GridCacheVersion(1, order, 1, otherClusterId), success);
+ }
+
+ /** Removes entry via {@link IgniteInternalCache#removeAllConflict(Map)}. */
+ private void removeConflict(String k, GridCacheVersion ver, boolean success) throws IgniteCheckedException {
+ CacheEntry<String, ConflictResolvableTestData> oldVal = cache.getEntry(k);
Review comment:
oldEntry?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheEntry;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Other cluster id. */
+ @Parameterized.Parameter(1)
+ public byte otherClusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, otherClusterId={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (byte otherClusterId : new byte[] {FIRST_CLUSTER_ID, THIRD_CLUSTER_ID})
+ params.add(new Object[] {mode, otherClusterId});
+
+ return params;
+ }
+
+ /** */
+ private static IgniteCache<String, ConflictResolvableTestData> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName).setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, ConflictResolvableTestData>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = key("UpdateFromOtherClusterWithoutConflict", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ putConflict(key, 2, true);
+
+ removeConflict(key, 3, true);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = key("UpdateClusterUpdateReorder", otherClusterId);
+
+ putConflict(key, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putConflict(key, 2, false);
+ putConflict(key, 1, false);
+
+ // Remove with the equal or lower order should fail.
Review comment:
should be ignored?
applicable for other comments
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheEntry;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Other cluster id. */
+ @Parameterized.Parameter(1)
+ public byte otherClusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, otherClusterId={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (byte otherClusterId : new byte[] {FIRST_CLUSTER_ID, THIRD_CLUSTER_ID})
+ params.add(new Object[] {mode, otherClusterId});
+
+ return params;
+ }
+
+ /** */
+ private static IgniteCache<String, ConflictResolvableTestData> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName).setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, ConflictResolvableTestData>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = key("UpdateFromOtherClusterWithoutConflict", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ putConflict(key, 2, true);
+
+ removeConflict(key, 3, true);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
Review comment:
Javadoc seems to be incorrect
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheEntry;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Other cluster id. */
+ @Parameterized.Parameter(1)
+ public byte otherClusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, otherClusterId={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (byte otherClusterId : new byte[] {FIRST_CLUSTER_ID, THIRD_CLUSTER_ID})
+ params.add(new Object[] {mode, otherClusterId});
+
+ return params;
+ }
+
+ /** */
+ private static IgniteCache<String, ConflictResolvableTestData> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName).setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, ConflictResolvableTestData>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = key("UpdateFromOtherClusterWithoutConflict", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ putConflict(key, 2, true);
+
+ removeConflict(key, 3, true);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = key("UpdateClusterUpdateReorder", otherClusterId);
+
+ putConflict(key, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putConflict(key, 2, false);
+ putConflict(key, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removeConflict(key, 2, false);
+ removeConflict(key, 1, false);
+
+ // Remove with the higher order should succeed.
+ putConflict(key, 3, true);
+
+ key = key("UpdateClusterUpdateReorder2", otherClusterId);
+
+ int order = 1;
+
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), true);
+
+ // Update with the equal or lower topVer should fail.
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower topVer should fail.
+ removeConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the higher topVer should succeed.
+ putConflict(key, new GridCacheVersion(3, order, 1, otherClusterId), true);
+
+ key = key("UpdateClusterUpdateReorder3", otherClusterId);
+
+ int topVer = 1;
+
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), true);
+
+ // Update with the equal or lower nodeOrder should fail.
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower nodeOrder should fail.
+ removeConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the higher nodeOrder should succeed.
+ putConflict(key, new GridCacheVersion(topVer, order, 3, otherClusterId), true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = key("UpdateThisClusterConflict0", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key);
+
+ // Conflict replicated update should fail.
+ putConflict(key, 2, false);
+
+ key = key("UpdateThisDCConflict1", otherClusterId);
+
+ putConflict(key, 3, true);
+
+ // Local update for other cluster entry should succeed.
+ put(key);
+
+ key = key("UpdateThisDCConflict2", otherClusterId);
+
+ put(key);
+
+ // Conflict replicated remove should fail.
+ removeConflict(key, 4, false);
+
+ key = key("UpdateThisDCConflict3", otherClusterId);
+
+ put(key);
+
+ // Conflict replicated update succeed only if resolved by field.
+ putConflict(key, 5, conflictResolveField() != null);
+ }
+
+ /** */
+ private void put(String key) {
+ ConflictResolvableTestData newVal = ConflictResolvableTestData.create();
+
+ CacheEntry<String, ConflictResolvableTestData> oldEntry = cache.getEntry(key);
+
+ cache.put(key, newVal);
+
+ CacheEntry<String, ConflictResolvableTestData> newEntry = cache.getEntry(key);
+
+ assertNull(((CacheEntryVersion)newEntry.version()).otherClusterVersion());
+ assertEquals(newVal, cache.get(key));
+
+ if (oldEntry != null)
+ assertTrue(((CacheEntryVersion)oldEntry.version()).order() < ((CacheEntryVersion)newEntry.version()).order());
+ }
+
+ /** Puts entry via {@link IgniteInternalCache#putAllConflict(Map)}. */
+ private void putConflict(String k, long order, boolean success) throws IgniteCheckedException {
+ putConflict(k, new GridCacheVersion(1, order, 1, otherClusterId), success);
+ }
+
+ /** Puts entry via {@link IgniteInternalCache#putAllConflict(Map)}. */
+ private void putConflict(String k, GridCacheVersion newVer, boolean success) throws IgniteCheckedException {
+ CacheEntry<String, ConflictResolvableTestData> oldVal = cache.getEntry(k);
+ ConflictResolvableTestData newVal = ConflictResolvableTestData.create();
+
+ KeyCacheObject key = new KeyCacheObjectImpl(k, null, cachex.context().affinity().partition(k));
+ CacheObject val = new CacheObjectImpl(client.binary().toBinary(newVal), null);
+
+ cachex.putAllConflict(singletonMap(key, new GridCacheDrInfo(val, newVer)));
+
+ if (success) {
+ assertEquals(newVer, ((CacheEntryVersion)cache.getEntry(k).version()).otherClusterVersion());
+ assertEquals(newVal, cache.get(k));
+ } else if (oldVal != null) {
+ assertEquals(oldVal.getValue(), cache.get(k));
+ assertEquals(oldVal.version(), cache.getEntry(k).version());
+ }
+ }
+
+ /** */
+ private void remove(String key) {
+ cache.remove(key);
Review comment:
assertion we have the value before the remove will be useful.
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheEntry;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Other cluster id. */
+ @Parameterized.Parameter(1)
+ public byte otherClusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, otherClusterId={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (byte otherClusterId : new byte[] {FIRST_CLUSTER_ID, THIRD_CLUSTER_ID})
+ params.add(new Object[] {mode, otherClusterId});
+
+ return params;
+ }
+
+ /** */
+ private static IgniteCache<String, ConflictResolvableTestData> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName).setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, ConflictResolvableTestData>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = key("UpdateFromOtherClusterWithoutConflict", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ putConflict(key, 2, true);
+
+ removeConflict(key, 3, true);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = key("UpdateClusterUpdateReorder", otherClusterId);
+
+ putConflict(key, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putConflict(key, 2, false);
+ putConflict(key, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removeConflict(key, 2, false);
+ removeConflict(key, 1, false);
+
+ // Remove with the higher order should succeed.
+ putConflict(key, 3, true);
+
+ key = key("UpdateClusterUpdateReorder2", otherClusterId);
+
+ int order = 1;
+
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), true);
+
+ // Update with the equal or lower topVer should fail.
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower topVer should fail.
+ removeConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the higher topVer should succeed.
+ putConflict(key, new GridCacheVersion(3, order, 1, otherClusterId), true);
+
+ key = key("UpdateClusterUpdateReorder3", otherClusterId);
+
+ int topVer = 1;
+
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), true);
+
+ // Update with the equal or lower nodeOrder should fail.
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower nodeOrder should fail.
+ removeConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the higher nodeOrder should succeed.
+ putConflict(key, new GridCacheVersion(topVer, order, 3, otherClusterId), true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = key("UpdateThisClusterConflict0", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key);
+
+ // Conflict replicated update should fail.
Review comment:
need to explain here that remove makes the comparison by conflictResolveField impossible.
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheEntry;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Other cluster id. */
+ @Parameterized.Parameter(1)
+ public byte otherClusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, otherClusterId={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (byte otherClusterId : new byte[] {FIRST_CLUSTER_ID, THIRD_CLUSTER_ID})
+ params.add(new Object[] {mode, otherClusterId});
+
+ return params;
+ }
+
+ /** */
+ private static IgniteCache<String, ConflictResolvableTestData> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName).setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, ConflictResolvableTestData>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = key("UpdateFromOtherClusterWithoutConflict", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ putConflict(key, 2, true);
+
+ removeConflict(key, 3, true);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = key("UpdateClusterUpdateReorder", otherClusterId);
+
+ putConflict(key, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putConflict(key, 2, false);
+ putConflict(key, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removeConflict(key, 2, false);
+ removeConflict(key, 1, false);
+
+ // Remove with the higher order should succeed.
+ putConflict(key, 3, true);
+
+ key = key("UpdateClusterUpdateReorder2", otherClusterId);
+
+ int order = 1;
+
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), true);
+
+ // Update with the equal or lower topVer should fail.
+ putConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower topVer should fail.
+ removeConflict(key, new GridCacheVersion(2, order, 1, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(1, order, 1, otherClusterId), false);
+
+ // Remove with the higher topVer should succeed.
+ putConflict(key, new GridCacheVersion(3, order, 1, otherClusterId), true);
+
+ key = key("UpdateClusterUpdateReorder3", otherClusterId);
+
+ int topVer = 1;
+
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), true);
+
+ // Update with the equal or lower nodeOrder should fail.
+ putConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ putConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the equal or lower nodeOrder should fail.
+ removeConflict(key, new GridCacheVersion(topVer, order, 2, otherClusterId), false);
+ removeConflict(key, new GridCacheVersion(topVer, order, 1, otherClusterId), false);
+
+ // Remove with the higher nodeOrder should succeed.
+ putConflict(key, new GridCacheVersion(topVer, order, 3, otherClusterId), true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = key("UpdateThisClusterConflict0", otherClusterId);
+
+ putConflict(key, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key);
+
+ // Conflict replicated update should fail.
+ putConflict(key, 2, false);
+
+ key = key("UpdateThisDCConflict1", otherClusterId);
+
+ putConflict(key, 3, true);
+
+ // Local update for other cluster entry should succeed.
+ put(key);
+
+ key = key("UpdateThisDCConflict2", otherClusterId);
+
+ put(key);
+
+ // Conflict replicated remove should fail.
+ removeConflict(key, 4, false);
+
+ key = key("UpdateThisDCConflict3", otherClusterId);
+
+ put(key);
+
+ // Conflict replicated update succeed only if resolved by field.
+ putConflict(key, 5, conflictResolveField() != null);
+ }
+
+ /** */
+ private void put(String key) {
+ ConflictResolvableTestData newVal = ConflictResolvableTestData.create();
+
+ CacheEntry<String, ConflictResolvableTestData> oldEntry = cache.getEntry(key);
+
+ cache.put(key, newVal);
+
+ CacheEntry<String, ConflictResolvableTestData> newEntry = cache.getEntry(key);
+
+ assertNull(((CacheEntryVersion)newEntry.version()).otherClusterVersion());
+ assertEquals(newVal, cache.get(key));
+
+ if (oldEntry != null)
+ assertTrue(((CacheEntryVersion)oldEntry.version()).order() < ((CacheEntryVersion)newEntry.version()).order());
+ }
+
+ /** Puts entry via {@link IgniteInternalCache#putAllConflict(Map)}. */
+ private void putConflict(String k, long order, boolean success) throws IgniteCheckedException {
+ putConflict(k, new GridCacheVersion(1, order, 1, otherClusterId), success);
+ }
+
+ /** Puts entry via {@link IgniteInternalCache#putAllConflict(Map)}. */
+ private void putConflict(String k, GridCacheVersion newVer, boolean success) throws IgniteCheckedException {
+ CacheEntry<String, ConflictResolvableTestData> oldVal = cache.getEntry(k);
Review comment:
should this be oldEntry?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652498207
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable oldResolveField;
+ Comparable newResolveField;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ oldResolveField = ((BinaryObject)oldVal).field(conflictResolveField);
+ newResolveField = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ oldResolveField = U.field(oldVal, conflictResolveField);
+ newResolveField = U.field(newVal, conflictResolveField);
+ }
+
+ if (oldResolveField == null || newResolveField == null)
+ return oldResolveField == null;
+
+ return oldResolveField.compareTo(newResolveField) < 0;
+ }
+ catch (Exception e) {
Review comment:
what exception do you expect here?
looks like this code should not throw anything
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER_ID},
+ {TRANSACTIONAL, THIRD_CLUSTER_ID},
+ {ATOMIC, FIRST_CLUSTER_ID},
+ {TRANSACTIONAL, FIRST_CLUSTER_ID},
+ });
+ }
+
+ /** */
+ private static IgniteCache<String, Data> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** Cluster have a greater priority that {@link #SECOND_CLUSTER_ID}. */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** Cluster have a lower priority that {@link #SECOND_CLUSTER_ID}. */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName)
+ .setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, Data>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = "UpdateFromOtherClusterWithoutConflict";
+
+ putx(key(key, clusterId), clusterId, 1, true);
+
+ putx(key(key, clusterId), clusterId, 2, true);
+
+ removex(key(key, clusterId), clusterId, 3, true);
+ }
+
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = "UpdateClusterUpdateReorder";
+
+ putx(key(key, clusterId), clusterId, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putx(key(key, clusterId), clusterId, 2, false);
+ putx(key(key, clusterId), clusterId, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removex(key(key, clusterId), clusterId, 2, false);
+ removex(key(key, clusterId), clusterId, 1, false);
+
+ // Remove with the higher order should succeed.
+ putx(key(key, clusterId), clusterId, 3, true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = "UpdateThisClusterConflict0";
+
+ putx(key(key, clusterId), clusterId, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key(key, clusterId));
+
+ // Conflict replicated update succeed only if cluster has a greater priority than this cluster.
+ putx(key(key, clusterId), clusterId, 2, clusterId == FIRST_CLUSTER_ID);
+
+ key = "UpdateThisDCConflict1";
+
+ putx(key(key, clusterId), clusterId, 3, true);
+
+ // Local update for other cluster entry should succeed.
+ put(key(key, clusterId));
+
+ key = "UpdateThisDCConflict2";
+
+ put(key(key, clusterId));
+
+ // Conflict replicated remove succeed only if DC has a greater priority than this DC.
+ removex(key(key, clusterId), clusterId, 4, clusterId == FIRST_CLUSTER_ID);
+
+ key = "UpdateThisDCConflict3";
+
+ put(key(key, clusterId));
+
+ // Conflict replicated update succeed only if DC has a greater priority than this DC.
+ putx(key(key, clusterId), clusterId, 5, clusterId == FIRST_CLUSTER_ID || conflictResolveField() != null);
+ }
+
+ /** */
+ private void put(String key) {
+ Data newVal = Data.create();
+
+ cache.put(key, newVal);
+
+ assertEquals(newVal, cache.get(key));
+ }
+
+ /** */
+ private void putx(String k, byte clusterId, long order, boolean expectSuccess) throws IgniteCheckedException {
+ Data oldVal = cache.get(k);
+ Data newVal = Data.create();
+
+ KeyCacheObject key = new KeyCacheObjectImpl(k, null, cachex.context().affinity().partition(k));
+ CacheObject val = new CacheObjectImpl(client.binary().toBinary(newVal), null);
+
+ cachex.putAllConflict(singletonMap(key, new GridCacheDrInfo(val, new GridCacheVersion(1, order, 1, clusterId))));
+
+ if (expectSuccess) {
+ assertTrue(cache.containsKey(k));
+
+ assertEquals(newVal, cache.get(k));
Review comment:
use getEntry to compare versions
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER_ID},
+ {TRANSACTIONAL, THIRD_CLUSTER_ID},
+ {ATOMIC, FIRST_CLUSTER_ID},
+ {TRANSACTIONAL, FIRST_CLUSTER_ID},
+ });
+ }
+
+ /** */
+ private static IgniteCache<String, Data> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx client;
+
+ /** Cluster have a greater priority that {@link #SECOND_CLUSTER_ID}. */
+ private static final byte FIRST_CLUSTER_ID = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER_ID = 2;
+
+ /** Cluster have a lower priority that {@link #SECOND_CLUSTER_ID}. */
+ private static final byte THIRD_CLUSTER_ID = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER_ID);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName)
+ .setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+ client = startClientGrid(2);
+
+ cache = client.createCache(new CacheConfiguration<String, Data>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = client.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = "UpdateFromOtherClusterWithoutConflict";
+
+ putx(key(key, clusterId), clusterId, 1, true);
+
+ putx(key(key, clusterId), clusterId, 2, true);
+
+ removex(key(key, clusterId), clusterId, 3, true);
+ }
+
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = "UpdateClusterUpdateReorder";
+
+ putx(key(key, clusterId), clusterId, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putx(key(key, clusterId), clusterId, 2, false);
+ putx(key(key, clusterId), clusterId, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removex(key(key, clusterId), clusterId, 2, false);
+ removex(key(key, clusterId), clusterId, 1, false);
+
+ // Remove with the higher order should succeed.
+ putx(key(key, clusterId), clusterId, 3, true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = "UpdateThisClusterConflict0";
+
+ putx(key(key, clusterId), clusterId, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key(key, clusterId));
+
+ // Conflict replicated update succeed only if cluster has a greater priority than this cluster.
+ putx(key(key, clusterId), clusterId, 2, clusterId == FIRST_CLUSTER_ID);
+
+ key = "UpdateThisDCConflict1";
+
+ putx(key(key, clusterId), clusterId, 3, true);
+
+ // Local update for other cluster entry should succeed.
+ put(key(key, clusterId));
+
+ key = "UpdateThisDCConflict2";
+
+ put(key(key, clusterId));
+
+ // Conflict replicated remove succeed only if DC has a greater priority than this DC.
+ removex(key(key, clusterId), clusterId, 4, clusterId == FIRST_CLUSTER_ID);
+
+ key = "UpdateThisDCConflict3";
+
+ put(key(key, clusterId));
+
+ // Conflict replicated update succeed only if DC has a greater priority than this DC.
+ putx(key(key, clusterId), clusterId, 5, clusterId == FIRST_CLUSTER_ID || conflictResolveField() != null);
+ }
+
+ /** */
+ private void put(String key) {
+ Data newVal = Data.create();
+
+ cache.put(key, newVal);
+
+ assertEquals(newVal, cache.get(key));
+ }
+
+ /** */
+ private void putx(String k, byte clusterId, long order, boolean expectSuccess) throws IgniteCheckedException {
+ Data oldVal = cache.get(k);
+ Data newVal = Data.create();
+
+ KeyCacheObject key = new KeyCacheObjectImpl(k, null, cachex.context().affinity().partition(k));
+ CacheObject val = new CacheObjectImpl(client.binary().toBinary(newVal), null);
+
+ cachex.putAllConflict(singletonMap(key, new GridCacheDrInfo(val, new GridCacheVersion(1, order, 1, clusterId))));
+
+ if (expectSuccess) {
+ assertTrue(cache.containsKey(k));
Review comment:
equality comparisons are enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655230551
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Caches. */
+ private final Map<Integer, IgniteInternalCache<BinaryObject, BinaryObject>> ignCaches = new HashMap<>();
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Maximum batch size. */
+ private final int maxBatchSize;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
Review comment:
It's not obvious from naming that one is inclusive but another is exclusive.
Lets, at least make both similar.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Executor service to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final ExecutorService execSvc;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+
+ execSvc = Executors.newFixedThreadPool(streamerCfg.getThreadCount(), new ThreadFactory() {
+ private final AtomicInteger cntr = new AtomicInteger();
+
+ @Override public Thread newThread(@NotNull Runnable r) {
+ Thread th = new Thread(r);
+
+ th.setName("applier-thread-" + cntr.getAndIncrement());
+
+ return th;
+ }
+ });
Review comment:
ExecutorSerice for a single run looks overcomplicated to me.
Why not just Thread.start/join?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
+ closed.set(true);
Review comment:
`stopped` seems to be more suitable.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/CdcEventsApplier.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.BooleanSupplier;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+
+/**
+ * Contains logic to process {@link ChangeDataCaptureEvent} and apply them to the provided by {@link #ignite()} cluster.
+ */
+public abstract class CdcEventsApplier {
+ /** Maximum batch size. */
+ private final int maxBatchSize;
+
+ /** Caches. */
+ private final Map<Integer, IgniteInternalCache<BinaryObject, BinaryObject>> ignCaches = new HashMap<>();
+
+ /** Update batch. */
+ private final Map<KeyCacheObject, GridCacheDrInfo> updBatch = new HashMap<>();
+
+ /** Remove batch. */
+ private final Map<KeyCacheObject, GridCacheVersion> rmvBatch = new HashMap<>();
+
+ /** */
+ private final BooleanSupplier hasUpdates = () -> !F.isEmpty(updBatch);
+
+ /** */
+ private final BooleanSupplier hasRemoves = () -> !F.isEmpty(rmvBatch);
+
+ /**
+ * @param maxBatchSize Maximum batch size.
+ */
+ public CdcEventsApplier(int maxBatchSize) {
+ this.maxBatchSize = maxBatchSize;
+ }
+
+ /**
+ * @param evts Events to process.
+ * @throws IgniteCheckedException If failed.
+ */
+ protected void apply(Iterable<ChangeDataCaptureEvent> evts) throws IgniteCheckedException {
+ IgniteInternalCache<BinaryObject, BinaryObject> currCache = null;
+
+ for (ChangeDataCaptureEvent evt : evts) {
+ IgniteInternalCache<BinaryObject, BinaryObject> cache = ignCaches.computeIfAbsent(evt.cacheId(), cacheId -> {
+ for (String cacheName : ignite().cacheNames()) {
+ if (CU.cacheId(cacheName) == cacheId) {
+ // IgniteEx#cachex(String) will return null if cache not initialized with regular Ignite#cache(String) call.
+ ignite().cache(cacheName);
+
+ return ignite().cachex(cacheName);
+ }
+ }
+
+ throw new IllegalStateException("Cache with id not found [cacheId=" + cacheId + ']');
+ });
+
+ if (cache != currCache) {
+ applyIf(currCache, hasUpdates, hasRemoves);
+
+ currCache = cache;
+ }
+
+ CacheEntryVersion order = evt.version();
+
+ KeyCacheObject key = new KeyCacheObjectImpl(evt.key(), null, evt.partition());
+
+ if (evt.value() != null) {
+ applyIf(currCache, () -> isApplyBatch(updBatch, key), hasRemoves);
+
+ CacheObject val = new CacheObjectImpl(evt.value(), null);
+
+ updBatch.put(key, new GridCacheDrInfo(val,
+ new GridCacheVersion(order.topologyVersion(), order.order(), order.nodeOrder(), order.clusterId())));
+ }
+ else {
+ applyIf(currCache, hasUpdates, () -> isApplyBatch(rmvBatch, key));
+
+ rmvBatch.put(key,
+ new GridCacheVersion(order.topologyVersion(), order.order(), order.nodeOrder(), order.clusterId()));
+ }
+ }
+
+ if (currCache != null)
+ applyIf(currCache, hasUpdates, hasRemoves);
+ }
+
+ /**
+ * Applies data from {@code updMap} or {@code rmvBatch} to Ignite if required.
Review comment:
What is `updMap`?
Frefactoring unfriendly comment found )
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
Review comment:
seems we're losing the reason here?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.CdcEventsApplier;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier extends CdcEventsApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
+
+ /** Caches ids to read. */
+ private final Set<Integer> caches;
+
+ /** Consumers. */
+ private final List<KafkaConsumer<Integer, byte[]>> cnsmrs = new ArrayList<>();
+
+ /** */
+ private final AtomicLong rcvdEvts = new AtomicLong();
+
+ /**
+ * @param ign Ignite instance.
+ * @param log Logger.
+ * @param kafkaProps Kafka properties.
+ * @param topic Topic name.
+ * @param kafkaPartFrom Read from partition.
+ * @param kafkaPartTo Read to partition.
+ * @param caches Cache ids.
+ * @param maxBatchSize Maximum batch size.
+ * @param closed Closed flag.
+ */
+ public KafkaToIgniteCdcStreamerApplier(
+ IgniteEx ign,
+ IgniteLogger log,
+ Properties kafkaProps,
+ String topic,
+ int kafkaPartFrom,
+ int kafkaPartTo,
+ Set<Integer> caches,
+ int maxBatchSize,
+ AtomicBoolean closed
+ ) {
+ super(maxBatchSize);
+
+ this.ign = ign;
+ this.kafkaProps = kafkaProps;
+ this.topic = topic;
+ this.kafkaPartFrom = kafkaPartFrom;
+ this.kafkaPartTo = kafkaPartTo;
+ this.caches = caches;
+ this.closed = closed;
+ this.log = log.getLogger(KafkaToIgniteCdcStreamerApplier.class);
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ U.setCurrentIgniteName(ign.name());
+
+ try {
+ for (int kafkaPart = kafkaPartFrom; kafkaPart < kafkaPartTo; kafkaPart++) {
+ KafkaConsumer<Integer, byte[]> cnsmr = new KafkaConsumer<>(kafkaProps);
+
+ cnsmr.assign(Collections.singleton(new TopicPartition(topic, kafkaPart)));
+
+ cnsmrs.add(cnsmr);
+ }
+
+ Iterator<KafkaConsumer<Integer, byte[]>> cnsmrIter = Collections.emptyIterator();
+
+ while (!closed.get()) {
+ if (!cnsmrIter.hasNext())
+ cnsmrIter = cnsmrs.iterator();
+
+ poll(cnsmrIter.next());
+ }
+ }
+ catch (WakeupException e) {
+ if (!closed.get())
+ log.error("Applier wakeup error!", e);
+ }
+ catch (Throwable e) {
+ log.error("Applier error!", e);
+
+ closed.set(true);
Review comment:
all other appliers should be waked up as well.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
+ closed.set(true);
+
+ appliers.forEach(U::closeQuiet);
Review comment:
any reason to call the method which sets `closed.set(true);` line after it set? %)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r651844592
##########
File path: modules/cdc-ext/modules/core/src/test/config/log4j-test.xml
##########
@@ -0,0 +1,145 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE log4j:configuration PUBLIC "-//APACHE//DTD LOG4J 1.2//EN"
+ "http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/xml/doc-files/log4j.dtd">
+<!--
+ Log4j configuration.
+-->
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">
+ <!--
+ Logs System.out messages to console.
+ -->
+ <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
+ <!-- Log to STDOUT. -->
+ <param name="Target" value="System.out"/>
+
+ <!-- Log from DEBUG and higher. -->
+ <param name="Threshold" value="DEBUG"/>
+
+ <!-- The default pattern: Date Priority [Category] Message\n -->
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
+ </layout>
+
+ <!-- Do not log beyond INFO level. -->
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="levelMin" value="DEBUG"/>
+ <param name="levelMax" value="INFO"/>
+ </filter>
+ </appender>
+
+ <!--
+ Logs all System.err messages to console.
+ -->
+ <appender name="CONSOLE_ERR" class="org.apache.log4j.ConsoleAppender">
+ <!-- Log to STDERR. -->
+ <param name="Target" value="System.err"/>
+
+ <!-- Log from WARN and higher. -->
+ <param name="Threshold" value="WARN"/>
+
+ <!-- The default pattern: Date Priority [Category] Message\n -->
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
+ </layout>
+ </appender>
+
+ <!--
+ Logs all output to specified file.
+ -->
+ <appender name="FILE" class="org.apache.log4j.RollingFileAppender">
+ <param name="Threshold" value="DEBUG"/>
+ <param name="File" value="${IGNITE_HOME}/work/log/ignite.log"/>
+ <param name="Append" value="true"/>
+ <param name="MaxFileSize" value="10MB"/>
+ <param name="MaxBackupIndex" value="10"/>
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
+ </layout>
+ </appender>
+
+ <!--
+ Uncomment to enable Ignite query execution debugging.
+ -->
+ <!--
+ <category name="org.apache.ignite.internal.processors.query">
+ <level value="DEBUG"/>
+ </category>
+ -->
+
+ <!--
+ Uncomment to enable Exchange, Rebalance and Partitions workflow debugging.
+ -->
+ <!--
+ <category name="org.apache.ignite.internal.processors.cache.distributed.dht.preloader">
+ <level value="DEBUG"/>
+ </category>
+
+ <category name="org.apache.ignite.internal.processors.cache.distributed.dht.topology">
+ <level value="DEBUG"/>
+ </category>
+ -->
+
+ <!--
+ Uncomment to enable debugging of partition counters.
+ -->
+ <!--
+ <category name="org.apache.ignite.internal.processors.cache.PartitionTxUpdateCounterDebugWrapper">
+ <level value="DEBUG"/>
+ </category>
+ -->
+
+ <!--
+ Uncomment to enable transactions debugging.
+ -->
+ <!--
+ <category name="org.apache.ignite.cache.msg.tx.prepare">
+ <level value="DEBUG"/>
+ </category>
+
+ <category name="org.apache.ignite.cache.msg.tx.finish">
+ <level value="DEBUG"/>
Review comment:
Could we get rid of commented sections?
##########
File path: modules/cdc-ext/modules/core/src/test/config/log4j-test.xml
##########
@@ -0,0 +1,145 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE log4j:configuration PUBLIC "-//APACHE//DTD LOG4J 1.2//EN"
Review comment:
Could we simplify/refactor this to make it as small and simple as possible?
##########
File path: modules/cdc-ext/modules/core/src/test/config/log4j-test.xml
##########
@@ -0,0 +1,145 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE log4j:configuration PUBLIC "-//APACHE//DTD LOG4J 1.2//EN"
+ "http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/xml/doc-files/log4j.dtd">
+<!--
+ Log4j configuration.
+-->
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">
+ <!--
+ Logs System.out messages to console.
+ -->
+ <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
+ <!-- Log to STDOUT. -->
+ <param name="Target" value="System.out"/>
+
+ <!-- Log from DEBUG and higher. -->
+ <param name="Threshold" value="DEBUG"/>
+
+ <!-- The default pattern: Date Priority [Category] Message\n -->
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
+ </layout>
+
+ <!-- Do not log beyond INFO level. -->
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="levelMin" value="DEBUG"/>
+ <param name="levelMax" value="INFO"/>
+ </filter>
+ </appender>
+
+ <!--
+ Logs all System.err messages to console.
+ -->
+ <appender name="CONSOLE_ERR" class="org.apache.log4j.ConsoleAppender">
+ <!-- Log to STDERR. -->
+ <param name="Target" value="System.err"/>
+
+ <!-- Log from WARN and higher. -->
+ <param name="Threshold" value="WARN"/>
+
+ <!-- The default pattern: Date Priority [Category] Message\n -->
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
+ </layout>
+ </appender>
+
+ <!--
+ Logs all output to specified file.
+ -->
+ <appender name="FILE" class="org.apache.log4j.RollingFileAppender">
+ <param name="Threshold" value="DEBUG"/>
+ <param name="File" value="${IGNITE_HOME}/work/log/ignite.log"/>
+ <param name="Append" value="true"/>
+ <param name="MaxFileSize" value="10MB"/>
+ <param name="MaxBackupIndex" value="10"/>
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
+ </layout>
+ </appender>
+
+ <!--
+ Uncomment to enable Ignite query execution debugging.
+ -->
+ <!--
+ <category name="org.apache.ignite.internal.processors.query">
+ <level value="DEBUG"/>
+ </category>
+ -->
+
+ <!--
+ Uncomment to enable Exchange, Rebalance and Partitions workflow debugging.
+ -->
+ <!--
+ <category name="org.apache.ignite.internal.processors.cache.distributed.dht.preloader">
+ <level value="DEBUG"/>
+ </category>
+
+ <category name="org.apache.ignite.internal.processors.cache.distributed.dht.topology">
+ <level value="DEBUG"/>
+ </category>
+ -->
+
+ <!--
+ Uncomment to enable debugging of partition counters.
+ -->
+ <!--
+ <category name="org.apache.ignite.internal.processors.cache.PartitionTxUpdateCounterDebugWrapper">
+ <level value="DEBUG"/>
+ </category>
+ -->
+
+ <!--
+ Uncomment to enable transactions debugging.
+ -->
+ <!--
+ <category name="org.apache.ignite.cache.msg.tx.prepare">
+ <level value="DEBUG"/>
+ </category>
+
+ <category name="org.apache.ignite.cache.msg.tx.finish">
+ <level value="DEBUG"/>
+ </category>
+
+ <category name="org.apache.ignite.cache.msg.tx.recovery">
+ <level value="DEBUG"/>
+ </category>
+ -->
+
+ <!-- Disable all open source debugging. -->
+ <category name="org">
+ <level value="INFO"/>
+ </category>
+
+ <category name="org.eclipse.jetty">
+ <level value="INFO"/>
+ </category>
+
+ <!-- Default settings. -->
+ <root>
+ <!-- Print at info by default. -->
+ <level value="INFO"/>
+
+ <!-- Append to file and console. -->
+ <appender-ref ref="FILE"/>
+ <appender-ref ref="CONSOLE"/>
Review comment:
Do we really need both?
##########
File path: modules/cdc-ext/modules/core/src/test/config/log4j-test.xml
##########
@@ -0,0 +1,145 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE log4j:configuration PUBLIC "-//APACHE//DTD LOG4J 1.2//EN"
+ "http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/xml/doc-files/log4j.dtd">
+<!--
+ Log4j configuration.
+-->
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">
+ <!--
+ Logs System.out messages to console.
+ -->
+ <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
+ <!-- Log to STDOUT. -->
+ <param name="Target" value="System.out"/>
+
+ <!-- Log from DEBUG and higher. -->
+ <param name="Threshold" value="DEBUG"/>
+
+ <!-- The default pattern: Date Priority [Category] Message\n -->
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
+ </layout>
+
+ <!-- Do not log beyond INFO level. -->
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="levelMin" value="DEBUG"/>
+ <param name="levelMax" value="INFO"/>
+ </filter>
+ </appender>
+
+ <!--
+ Logs all System.err messages to console.
+ -->
+ <appender name="CONSOLE_ERR" class="org.apache.log4j.ConsoleAppender">
+ <!-- Log to STDERR. -->
+ <param name="Target" value="System.err"/>
+
+ <!-- Log from WARN and higher. -->
+ <param name="Threshold" value="WARN"/>
+
+ <!-- The default pattern: Date Priority [Category] Message\n -->
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
+ </layout>
+ </appender>
+
+ <!--
+ Logs all output to specified file.
+ -->
+ <appender name="FILE" class="org.apache.log4j.RollingFileAppender">
+ <param name="Threshold" value="DEBUG"/>
Review comment:
Do we really need to log debug?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ o = ((BinaryObject)oldVal).field(conflictResolveField);
+ n = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ o = U.field(oldVal, conflictResolveField);
+ n = U.field(newVal, conflictResolveField);
+ }
+
+ if (o == null || n == null)
+ return o == null;
+
+ return o.compareTo(n) < 0;
+ }
+ catch (Exception e) {
+ log.error("Error while resolving replication conflict with field '" +
+ conflictResolveField + "'.\nConflict resolve with the field disabled.", e);
+
+ conflictResolveFieldEnabled = false;
+ }
+ }
+ }
+
+ // Cluster with the lower ID have biggest priority(e.g. first cluster is main).
+ return newEntry.dataCenterId() < oldEntry.dataCenterId();
Review comment:
Warn message with the key should be also generated here since the decision made by fuzzy logic.
BTW, do we able to mention keys in logs in theory? How about PCI-DSS?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.checkCRC;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.generateSingleData;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER},
+ {TRANSACTIONAL, THIRD_CLUSTER},
+ {ATOMIC, FIRST_CLUSTER},
+ {TRANSACTIONAL, FIRST_CLUSTER},
+ });
+ }
+
+ /** */
+ private static IgniteCache<String, Data> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx cli;
+
+ /** Cluster have a greater priority that {@link #SECOND_CLUSTER}. */
+ private static final byte FIRST_CLUSTER = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER = 2;
+
+ /** Cluster have a lower priority that {@link #SECOND_CLUSTER}. */
+ private static final byte THIRD_CLUSTER = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName)
+ .setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+ cli = startClientGrid(2);
+
+ cache = cli.createCache(new CacheConfiguration<String, Data>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = cli.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = "UpdateFromOtherClusterWithoutConflict";
+
+ putx(key(key, clusterId), clusterId, 1, true);
+
+ putx(key(key, clusterId), clusterId, 2, true);
+
+ removex(key(key, clusterId), clusterId, 3, true);
+ }
+
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = "UpdateClusterUpdateReorder";
+
+ putx(key(key, clusterId), clusterId, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putx(key(key, clusterId), clusterId, 2, false);
+ putx(key(key, clusterId), clusterId, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removex(key(key, clusterId), clusterId, 2, false);
+ removex(key(key, clusterId), clusterId, 1, false);
+
+ // Remove with the higher order should succeed.
+ putx(key(key, clusterId), clusterId, 3, true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = "UpdateThisClusterConflict0";
+
+ putx(key(key, clusterId), clusterId, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key(key, clusterId));
+
+ // Conflict replicated update succeed only if cluster has a greater priority than this cluster.
+ putx(key(key, clusterId), clusterId, 2, clusterId == FIRST_CLUSTER);
+
+ key = "UpdateThisDCConflict1";
+
+ putx(key(key, clusterId), clusterId, 3, true);
+
+ // Local update for other cluster entry should succeed.
+ put(key(key, clusterId));
+
+ key = "UpdateThisDCConflict2";
+
+ put(key(key, clusterId));
+
+ // Conflict replicated remove succeed only if DC has a greater priority than this DC.
+ removex(key(key, clusterId), clusterId, 4, clusterId == FIRST_CLUSTER);
+
+ key = "UpdateThisDCConflict3";
+
+ put(key(key, clusterId));
+
+ // Conflict replicated update succeed only if DC has a greater priority than this DC.
+ putx(key(key, clusterId), clusterId, 5, clusterId == FIRST_CLUSTER || conflictResolveField() != null);
+ }
+
+ /** */
+ private void put(String key) {
+ Data newVal = generateSingleData(1);
+
+ cache.put(key, newVal);
+
+ assertEquals(newVal, cache.get(key));
+
+ checkCRC(cache.get(key), 1);
+ }
+
+ /** */
+ private void putx(String k, byte clusterId, long order, boolean expectSuccess) throws IgniteCheckedException {
+ Data oldVal = cache.get(k);
+ Data newVal = generateSingleData(1);
+
+ KeyCacheObject key = new KeyCacheObjectImpl(k, null, cachex.context().affinity().partition(k));
+ CacheObject val = new CacheObjectImpl(cli.binary().toBinary(newVal), null);
+
+ cachex.putAllConflict(singletonMap(key, new GridCacheDrInfo(val, new GridCacheVersion(1, order, 1, clusterId))));
+
+ if (expectSuccess) {
Review comment:
seems you should just check the val and version equality to expected?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.checkCRC;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.generateSingleData;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER},
+ {TRANSACTIONAL, THIRD_CLUSTER},
+ {ATOMIC, FIRST_CLUSTER},
+ {TRANSACTIONAL, FIRST_CLUSTER},
Review comment:
how about
```
loop ...
loop ...
```
here and at others?
##########
File path: modules/cdc-ext/modules/core/src/test/config/tests.properties
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
Review comment:
Could we simplify/refactor this to make it as small and simple as possible?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ o = ((BinaryObject)oldVal).field(conflictResolveField);
+ n = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ o = U.field(oldVal, conflictResolveField);
+ n = U.field(newVal, conflictResolveField);
+ }
+
+ if (o == null || n == null)
Review comment:
Should this case produce at least a warning?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.checkCRC;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.generateSingleData;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER},
+ {TRANSACTIONAL, THIRD_CLUSTER},
+ {ATOMIC, FIRST_CLUSTER},
+ {TRANSACTIONAL, FIRST_CLUSTER},
+ });
+ }
+
+ /** */
+ private static IgniteCache<String, Data> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx cli;
+
+ /** Cluster have a greater priority that {@link #SECOND_CLUSTER}. */
+ private static final byte FIRST_CLUSTER = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER = 2;
+
+ /** Cluster have a lower priority that {@link #SECOND_CLUSTER}. */
+ private static final byte THIRD_CLUSTER = 3;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> pluginCfg = new CacheVersionConflictResolverPluginProvider<>();
+
+ pluginCfg.setClusterId(SECOND_CLUSTER);
+ pluginCfg.setCaches(new HashSet<>(Collections.singleton(DEFAULT_CACHE_NAME)));
+ pluginCfg.setConflictResolveField(conflictResolveField());
+
+ return super.getConfiguration(igniteInstanceName)
+ .setPluginProviders(pluginCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ startGrid(1);
+ cli = startClientGrid(2);
+
+ cache = cli.createCache(new CacheConfiguration<String, Data>(DEFAULT_CACHE_NAME).setAtomicityMode(cacheMode));
+ cachex = cli.cachex(DEFAULT_CACHE_NAME);
+ }
+
+ /** Tests that regular cache operations works with the conflict resolver when there is no update conflicts. */
+ @Test
+ public void testSimpleUpdates() {
+ String key = "UpdatesWithoutConflict";
+
+ put(key);
+
+ put(key);
+
+ remove(key);
+ }
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesFromOtherClusterWithoutConflict() throws Exception {
+ String key = "UpdateFromOtherClusterWithoutConflict";
+
+ putx(key(key, clusterId), clusterId, 1, true);
+
+ putx(key(key, clusterId), clusterId, 2, true);
+
+ removex(key(key, clusterId), clusterId, 3, true);
+ }
+
+
+ /**
+ * Tests that {@code IgniteInternalCache#*AllConflict} cache operations works with the conflict resolver
+ * when there is no update conflicts.
+ */
+ @Test
+ public void testUpdatesReorderFromOtherCluster() throws Exception {
+ String key = "UpdateClusterUpdateReorder";
+
+ putx(key(key, clusterId), clusterId, 2, true);
+
+ // Update with the equal or lower order should fail.
+ putx(key(key, clusterId), clusterId, 2, false);
+ putx(key(key, clusterId), clusterId, 1, false);
+
+ // Remove with the equal or lower order should fail.
+ removex(key(key, clusterId), clusterId, 2, false);
+ removex(key(key, clusterId), clusterId, 1, false);
+
+ // Remove with the higher order should succeed.
+ putx(key(key, clusterId), clusterId, 3, true);
+ }
+
+ /** Tests cache operations for entry replicated from another cluster. */
+ @Test
+ public void testUpdatesConflict() throws Exception {
+ String key = "UpdateThisClusterConflict0";
+
+ putx(key(key, clusterId), clusterId, 1, true);
+
+ // Local remove for other cluster entry should succeed.
+ remove(key(key, clusterId));
+
+ // Conflict replicated update succeed only if cluster has a greater priority than this cluster.
+ putx(key(key, clusterId), clusterId, 2, clusterId == FIRST_CLUSTER);
+
+ key = "UpdateThisDCConflict1";
+
+ putx(key(key, clusterId), clusterId, 3, true);
+
+ // Local update for other cluster entry should succeed.
+ put(key(key, clusterId));
+
+ key = "UpdateThisDCConflict2";
+
+ put(key(key, clusterId));
+
+ // Conflict replicated remove succeed only if DC has a greater priority than this DC.
+ removex(key(key, clusterId), clusterId, 4, clusterId == FIRST_CLUSTER);
+
+ key = "UpdateThisDCConflict3";
+
+ put(key(key, clusterId));
+
+ // Conflict replicated update succeed only if DC has a greater priority than this DC.
+ putx(key(key, clusterId), clusterId, 5, clusterId == FIRST_CLUSTER || conflictResolveField() != null);
+ }
+
+ /** */
+ private void put(String key) {
+ Data newVal = generateSingleData(1);
+
+ cache.put(key, newVal);
+
+ assertEquals(newVal, cache.get(key));
+
+ checkCRC(cache.get(key), 1);
+ }
+
+ /** */
+ private void putx(String k, byte clusterId, long order, boolean expectSuccess) throws IgniteCheckedException {
+ Data oldVal = cache.get(k);
+ Data newVal = generateSingleData(1);
+
+ KeyCacheObject key = new KeyCacheObjectImpl(k, null, cachex.context().affinity().partition(k));
+ CacheObject val = new CacheObjectImpl(cli.binary().toBinary(newVal), null);
+
+ cachex.putAllConflict(singletonMap(key, new GridCacheDrInfo(val, new GridCacheVersion(1, order, 1, clusterId))));
+
+ if (expectSuccess) {
+ assertTrue(cache.containsKey(k));
+
+ assertEquals(newVal, cache.get(k));
+
+ checkCRC(cache.get(k), newVal.getIter());
Review comment:
looks overcomplicated,
why not just equals?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.checkCRC;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.generateSingleData;
Review comment:
Please get rid of cross-tests imports
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.checkCRC;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.generateSingleData;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER},
+ {TRANSACTIONAL, THIRD_CLUSTER},
+ {ATOMIC, FIRST_CLUSTER},
+ {TRANSACTIONAL, FIRST_CLUSTER},
+ });
+ }
+
+ /** */
+ private static IgniteCache<String, Data> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx cli;
+
+ /** Cluster have a greater priority that {@link #SECOND_CLUSTER}. */
+ private static final byte FIRST_CLUSTER = 1;
+
+ /** */
+ private static final byte SECOND_CLUSTER = 2;
+
+ /** Cluster have a lower priority that {@link #SECOND_CLUSTER}. */
+ private static final byte THIRD_CLUSTER = 3;
Review comment:
*_CLUSTER_ID
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/CacheConflictOperationsTest.java
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.Data;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.checkCRC;
+import static org.apache.ignite.cdc.kafka.CdcKafkaReplicationTest.generateSingleData;
+
+/**
+ * Cache conflict operations test.
+ */
+@RunWith(Parameterized.class)
+public class CacheConflictOperationsTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** Cluster id. */
+ @Parameterized.Parameter(1)
+ public byte clusterId;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0}, clusterId={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, THIRD_CLUSTER},
+ {TRANSACTIONAL, THIRD_CLUSTER},
+ {ATOMIC, FIRST_CLUSTER},
+ {TRANSACTIONAL, FIRST_CLUSTER},
+ });
+ }
+
+ /** */
+ private static IgniteCache<String, Data> cache;
+
+ /** */
+ private static IgniteInternalCache<BinaryObject, BinaryObject> cachex;
+
+ /** */
+ private static IgniteEx cli;
Review comment:
client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r651964691
##########
File path: modules/cdc-ext/modules/core/src/test/config/log4j-test.xml
##########
@@ -0,0 +1,145 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE log4j:configuration PUBLIC "-//APACHE//DTD LOG4J 1.2//EN"
Review comment:
AFAICU these files exists in each extension module is just a copy of `log4j-test.xml` from core module sources.
It seems to me, we should refactor to get rid of all of them.
But, this is out of the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652425982
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ o = ((BinaryObject)oldVal).field(conflictResolveField);
+ n = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ o = U.field(oldVal, conflictResolveField);
+ n = U.field(newVal, conflictResolveField);
+ }
+
+ if (o == null || n == null)
+ return o == null;
+
+ return o.compareTo(n) < 0;
+ }
+ catch (Exception e) {
+ log.error("Error while resolving replication conflict with field '" +
+ conflictResolveField + "'.\nConflict resolve with the field disabled.", e);
+
+ conflictResolveFieldEnabled = false;
+ }
+ }
+ }
+
+ // Cluster with the lower ID have biggest priority(e.g. first cluster is main).
+ return newEntry.dataCenterId() < oldEntry.dataCenterId();
Review comment:
> Warn message with the key should be also generated here since the decision made by fuzzy logic.
Conflict resolver is extremely hot code so I don't think we can insert logging here.
> BTW, do we able to mention keys in logs in theory?
Yes, please, take a look at `IgniteSystemProperties#IGNITE_TO_STRING_INCLUDE_SENSITIVE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648033254
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/KafkaUtils.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.DescribeTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.common.errors.TopicExistsException;
+import org.apache.kafka.common.errors.UnknownTopicOrPartitionException;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/** Kafka Utils. */
+public class KafkaUtils {
+ /** Default kafka request timeout. */
+ public static final int TIMEOUT_MIN = 1;
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_NUM_PARTITIONS = "ignite.to.kafka.numpartitions";
Review comment:
```
public static final String IGNITE_TO_KAFKA_PART_COUNT = "ignite.to.kafka.partitions.count";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r649006352
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/conflictplugin/CacheConflictResolutionManagerImpl.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictplugin;
Review comment:
org.apache.ignite.cdc.conflict(s,resolve)
##########
File path: modules/cdc-to-kafka-ext/src/test/java/org/apache/ignite/cdc/KafkaToIgniteLoaderTest.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.KafkaToIgniteLoader.loadKafkaToIgniteStreamer;
+import static org.apache.ignite.testframework.GridTestUtils.assertThrows;
+import static org.apache.ignite.testframework.GridTestUtils.assertThrowsWithCause;
+
+/** Tests load {@link KafkaToIgniteCdcStreamer} from Srping xml file. */
+public class KafkaToIgniteLoaderTest extends GridCommonAbstractTest {
+ /** */
+ @Test
+ public void testLoadConfig() throws Exception {
+ assertThrows(
+ null,
+ () -> loadKafkaToIgniteStreamer("kafka-to-ignite-2.xml"),
+ IgniteCheckedException.class,
+ "Exact 1 IgniteConfiguration should be defined. Found 2"
+ );
+
+ assertThrowsWithCause(
+ () -> loadKafkaToIgniteStreamer("kafka-to-ignite-3.xml"),
Review comment:
the problem should be explained at cfg name, not just 3
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/IgniteToKafkaCdcStreamer.java
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link KafkaToIgniteCdcStreamer} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see KafkaToIgniteCdcStreamer
+ * @see CacheVersionConflictResolverImpl
+ */
+public class IgniteToKafkaCdcStreamer implements ChangeDataCaptureConsumer {
+ /** Default kafka request timeout. */
+ public static final int DFLT_REQ_TIMEOUT_MIN = 1;
+
+ /** Log. */
+ @LoggerResource
+ private IgniteLogger log;
+
+ /** Kafka producer to stream events. */
+ private KafkaProducer<Integer, ChangeDataCaptureEvent> producer;
+
+ /** Handle only primary entry flag. */
+ private final boolean onlyPrimary;
+
+ /** Topic name. */
+ private final String topic;
+
+ /** Kafka topic partitions count. */
+ private final int kafkaParts;
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Cache IDs. */
+ private final Set<Integer> cachesIds;
+
+ /** Max batch size. */
+ private final int maxBatchSize;
+
+ /** Count of sent messages. */
+ private long msgCnt;
+
+ /**
+ * @param topic Topic name.
+ * @param kafkaParts Kafka partitions count.
+ * @param caches Cache names.
+ * @param maxBatchSize Maximum count of concurrently.
Review comment:
> Maximum count of concurrently.
Whan this means?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/conflictplugin/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictplugin;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * Note, values of this field must implement {@link Comparable} interface.
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict[key=" + newEntry.key() + ",fromCluster=" + newEntry.dataCenterId()
+ + ",toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (newEntry.dataCenterId() == clusterId) // Update made on the same DC.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.order() > oldEntry.order(); // New version from the same DC.
Review comment:
version should be checked instead.
##########
File path: modules/cdc-to-kafka-ext/src/test/java/org/apache/ignite/cdc/KafkaToIgniteLoaderTest.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.KafkaToIgniteLoader.loadKafkaToIgniteStreamer;
+import static org.apache.ignite.testframework.GridTestUtils.assertThrows;
+import static org.apache.ignite.testframework.GridTestUtils.assertThrowsWithCause;
+
+/** Tests load {@link KafkaToIgniteCdcStreamer} from Srping xml file. */
+public class KafkaToIgniteLoaderTest extends GridCommonAbstractTest {
+ /** */
+ @Test
+ public void testLoadConfig() throws Exception {
+ assertThrows(
+ null,
+ () -> loadKafkaToIgniteStreamer("kafka-to-ignite-2.xml"),
+ IgniteCheckedException.class,
+ "Exact 1 IgniteConfiguration should be defined. Found 2"
+ );
+
+ assertThrowsWithCause(
+ () -> loadKafkaToIgniteStreamer("kafka-to-ignite-3.xml"),
+ IgniteCheckedException.class
+ );
Review comment:
Exception message specification missed
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/IgniteToKafkaCdcStreamer.java
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link KafkaToIgniteCdcStreamer} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see KafkaToIgniteCdcStreamer
+ * @see CacheVersionConflictResolverImpl
+ */
+public class IgniteToKafkaCdcStreamer implements ChangeDataCaptureConsumer {
+ /** Default kafka request timeout. */
+ public static final int DFLT_REQ_TIMEOUT_MIN = 1;
Review comment:
DFLT_REQ_TIMEOUT
as I told you before, the resolution should be specified at javadoc.
##########
File path: modules/cdc-to-kafka-ext/src/test/java/org/apache/ignite/cdc/KafkaToIgniteLoaderTest.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static org.apache.ignite.cdc.KafkaToIgniteLoader.loadKafkaToIgniteStreamer;
+import static org.apache.ignite.testframework.GridTestUtils.assertThrows;
+import static org.apache.ignite.testframework.GridTestUtils.assertThrowsWithCause;
+
+/** Tests load {@link KafkaToIgniteCdcStreamer} from Srping xml file. */
+public class KafkaToIgniteLoaderTest extends GridCommonAbstractTest {
+ /** */
+ @Test
+ public void testLoadConfig() throws Exception {
+ assertThrows(
+ null,
+ () -> loadKafkaToIgniteStreamer("kafka-to-ignite-2.xml"),
Review comment:
the problem should be explained at cfg name, not just 2
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/IgniteToKafkaCdcStreamer.java
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link KafkaToIgniteCdcStreamer} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see KafkaToIgniteCdcStreamer
+ * @see CacheVersionConflictResolverImpl
+ */
+public class IgniteToKafkaCdcStreamer implements ChangeDataCaptureConsumer {
+ /** Default kafka request timeout. */
+ public static final int DFLT_REQ_TIMEOUT_MIN = 1;
+
+ /** Log. */
+ @LoggerResource
+ private IgniteLogger log;
+
+ /** Kafka producer to stream events. */
+ private KafkaProducer<Integer, ChangeDataCaptureEvent> producer;
+
+ /** Handle only primary entry flag. */
+ private final boolean onlyPrimary;
+
+ /** Topic name. */
+ private final String topic;
+
+ /** Kafka topic partitions count. */
+ private final int kafkaParts;
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Cache IDs. */
+ private final Set<Integer> cachesIds;
+
+ /** Max batch size. */
+ private final int maxBatchSize;
+
+ /** Count of sent messages. */
+ private long msgCnt;
+
+ /**
+ * @param topic Topic name.
+ * @param kafkaParts Kafka partitions count.
+ * @param caches Cache names.
+ * @param maxBatchSize Maximum count of concurrently.
+ * @param onlyPrimary If {@code true} then stream only events from primaries.
+ * @param kafkaProps Kafka properties.
+ */
+ public IgniteToKafkaCdcStreamer(
+ String topic,
+ int kafkaParts,
+ Set<String> caches,
+ int maxBatchSize,
+ boolean onlyPrimary,
+ Properties kafkaProps
+ ) {
+ assert caches != null && !caches.isEmpty();
+
+ this.topic = topic;
+ this.kafkaParts = kafkaParts;
+ this.onlyPrimary = onlyPrimary;
+ this.kafkaProps = kafkaProps;
+ this.maxBatchSize = maxBatchSize;
+
+ cachesIds = caches.stream()
+ .mapToInt(CU::cacheId)
+ .boxed()
+ .collect(Collectors.toSet());
+ }
+
+ /** {@inheritDoc} */
+ @Override public boolean onEvents(Iterator<ChangeDataCaptureEvent> evts) {
+ List<Future<RecordMetadata>> futs = new ArrayList<>();
+
+ while (evts.hasNext() && futs.size() <= maxBatchSize) {
Review comment:
should be `<`?
##########
File path: modules/cdc-to-kafka-ext/src/test/java/org/apache/ignite/cdc/CdcKafkaReplicationTest.java
##########
@@ -0,0 +1,519 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.cdc.serde.JavaObjectSerializer;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.processors.cache.persistence.wal.crc.FastCrc;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.AbstractChangeDataCaptureTest.KEYS_CNT;
Review comment:
please avoid such usages when the test is not an inheritor
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/serde/JavaObjectSerializer.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.serde;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import org.apache.kafka.common.serialization.Serializer;
+
+/** Java object serializer. */
+public class JavaObjectSerializer implements Serializer<Serializable> {
Review comment:
use IgniteUtils#toBytes instead
##########
File path: modules/cdc-to-kafka-ext/src/test/java/org/apache/ignite/cdc/CdcKafkaReplicationTest.java
##########
@@ -0,0 +1,523 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.cdc.serde.JavaObjectSerializer;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.processors.cache.persistence.wal.crc.FastCrc;
+import org.apache.ignite.internal.processors.resource.GridSpringResourceContext;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cdc.AbstractChangeDataCaptureTest.KEYS_CNT;
+import static org.apache.ignite.cdc.IgniteToKafkaCdcStreamer.DFLT_REQ_TIMEOUT_MIN;
+import static org.apache.ignite.cdc.KafkaToIgniteCdcStreamerConfiguration.DFLT_PARTS;
+import static org.apache.ignite.cdc.KafkaToIgniteCdcStreamerConfiguration.DFLT_TOPIC;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/**
+ * Tests for kafka replication.
+ */
+@RunWith(Parameterized.class)
+public class CdcKafkaReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ return Arrays.asList(new Object[][] {
+ {ATOMIC, 0},
+ {ATOMIC, 1},
+ {TRANSACTIONAL, 0},
+ {TRANSACTIONAL, 1}
+ });
+ }
+
+ /** */
+ public static final String SRC_DEST_TOPIC = "source-dest";
+
+ /** */
+ public static final String DEST_SRC_TOPIC = "dest-source";
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int BOTH_EXISTS = 1;
+
+ /** */
+ public static final int BOTH_REMOVED = 2;
+
+ /** */
+ public static final int ANY_SAME_STATE = 3;
+
+ /** */
+ public static final AtomicLong REQUEST_ID = new AtomicLong();
+
+ /** */
+ private static Properties props;
+
+ /** */
+ private static IgniteEx[] srcCluster;
+
+ /** */
+ private static IgniteConfiguration srcClusterCliCfg;
+
+ /** */
+ private static IgniteEx[] destCluster;
+
+ /** */
+ private static IgniteConfiguration destClusterCliCfg;
+
+ /** */
+ @ClassRule
+ public static KafkaContainer kafka = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.4.3"));
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ private static final ThreadLocal<FastCrc> crc = new ThreadLocal<>().withInitial(FastCrc::new);
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setLocalPortRange(DFLT_PORT_RANGE)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort)
+ .setLocalPortRange(DFLT_PORT_RANGE))
+ .setPluginProviders(cfgPlugin);
+
+ if (!cfg.isClientMode()) {
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ kafka.start();
+
+ if (props == null) {
+ props = new Properties();
+
+ props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
+
+ props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class.getName());
+ props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JavaObjectSerializer.class.getName());
+
+ props.put(ConsumerConfig.GROUP_ID_CONFIG, "kafka-to-ignite-applier");
+ props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
+ props.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 10_000);
+ }
+
+ createTopic(DFLT_TOPIC, DFLT_PARTS, props);
+ createTopic(SRC_DEST_TOPIC, DFLT_PARTS, props);
+ createTopic(DEST_SRC_TOPIC, DFLT_PARTS, props);
+
+ srcCluster = new IgniteEx[] {
+ startGrid(1),
+ startGrid(2)
+ };
+
+ srcClusterCliCfg = optimize(getConfiguration("src-cluster-client").setClientMode(true));
+
+ srcCluster[0].cluster().state(ACTIVE);
+ srcCluster[0].cluster().tag("source");
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ commPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = new IgniteEx[] {
+ startGrid(4),
+ startGrid(5)
+ };
+
+ destClusterCliCfg = optimize(getConfiguration("dest-cluster-client").setClientMode(true));
+
+ assertFalse("source".equals(destCluster[0].cluster().tag()));
+
+ destCluster[0].cluster().state(ACTIVE);
+ destCluster[0].cluster().tag("destination");
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ props = null;
+
+ stopAllGrids();
+
+ cleanPersistenceDir();
+
+ kafka.stop();
+ }
+
+ /** */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ IgniteInternalFuture<?> fut1 = igniteToKafka(srcCluster[0], DFLT_TOPIC, AP_CACHE);
+ IgniteInternalFuture<?> fut2 = igniteToKafka(srcCluster[1], DFLT_TOPIC, AP_CACHE);
+
+ try {
+ IgniteCache<Integer, Data> destCache = destCluster[0].createCache(AP_CACHE);
+
+ destCache.put(1, new Data(null, 0, 1, REQUEST_ID.incrementAndGet()));
+ destCache.remove(1);
+
+ runAsync(generateData("cache-1", srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT), 1));
+ runAsync(generateData(AP_CACHE, srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT), 1));
+
+ KafkaToIgniteCdcStreamerConfiguration cfg = new KafkaToIgniteCdcStreamerConfiguration();
+
+ cfg.setCacheNames(Collections.singletonList(AP_CACHE));
+
+ IgniteInternalFuture<?> k2iFut =
+ runAsync(new KafkaToIgniteCdcStreamer(destClusterCliCfg, props, cfg));
+
+ try {
+ IgniteCache<Integer, Data> srcCache = srcCluster[srcCluster.length - 1].getOrCreateCache(AP_CACHE);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, BOTH_EXISTS, 1, fut1, fut2, k2iFut);
+
+ IntStream.range(0, KEYS_CNT).forEach(srcCache::remove);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, BOTH_REMOVED, 1, fut1, fut2, k2iFut);
+ }
+ finally {
+ k2iFut.cancel();
+ }
+ }
+ finally {
+ fut1.cancel();
+ fut2.cancel();
+ }
+ }
+
+ /** */
+ @Test
+ public void testActiveActiveReplication() throws Exception {
+ IgniteCache<Integer, Data> srcCache = srcCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+ IgniteCache<Integer, Data> destCache = destCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, srcCluster[srcCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 == 0), 1));
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, destCluster[destCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 != 0), 1));
+
+ IgniteInternalFuture<?> cdcSrcFut1 = igniteToKafka(srcCluster[0], SRC_DEST_TOPIC, ACTIVE_ACTIVE_CACHE);
+ IgniteInternalFuture<?> cdcSrcFut2 = igniteToKafka(srcCluster[1], SRC_DEST_TOPIC, ACTIVE_ACTIVE_CACHE);
+ IgniteInternalFuture<?> cdcDestFut1 = igniteToKafka(destCluster[0], DEST_SRC_TOPIC, ACTIVE_ACTIVE_CACHE);
+ IgniteInternalFuture<?> cdcDestFut2 = igniteToKafka(destCluster[1], DEST_SRC_TOPIC, ACTIVE_ACTIVE_CACHE);
+
+ try {
+ KafkaToIgniteCdcStreamerConfiguration cfg1 = new KafkaToIgniteCdcStreamerConfiguration();
+ KafkaToIgniteCdcStreamerConfiguration cfg2 = new KafkaToIgniteCdcStreamerConfiguration();
+
+ cfg1.setTopic(SRC_DEST_TOPIC);
+ cfg1.setCacheNames(Collections.singletonList(ACTIVE_ACTIVE_CACHE));
+
+ cfg2.setTopic(DEST_SRC_TOPIC);
+ cfg2.setCacheNames(Collections.singletonList(ACTIVE_ACTIVE_CACHE));
+
+ IgniteInternalFuture<?> k2iFut1 = runAsync(new KafkaToIgniteCdcStreamer(destClusterCliCfg, props, cfg1));
+ IgniteInternalFuture<?> k2iFut2 = runAsync(new KafkaToIgniteCdcStreamer(srcClusterCliCfg, props, cfg2));
+
+ try {
+ waitForSameData(srcCache, destCache, KEYS_CNT, BOTH_EXISTS, 1,
+ cdcDestFut1, cdcDestFut2, cdcDestFut1, cdcDestFut2, k2iFut1, k2iFut2);
+
+ for (int i = 0; i < KEYS_CNT; i++) {
+ srcCache.put(i, generateSingleData(2));
+ destCache.put(i, generateSingleData(2));
+ }
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, ANY_SAME_STATE, 2,
+ cdcDestFut1, cdcDestFut2, cdcDestFut1, cdcDestFut2, k2iFut1, k2iFut2);
+ }
+ finally {
+ k2iFut1.cancel();
+ k2iFut2.cancel();
+ }
+ }
+ finally {
+ cdcSrcFut1.cancel();
+ cdcSrcFut2.cancel();
+ cdcDestFut1.cancel();
+ cdcDestFut2.cancel();
+ }
+ }
+
+ /** */
+ public void waitForSameData(
+ IgniteCache<Integer, Data> src,
+ IgniteCache<Integer, Data> dest,
+ int keysCnt,
+ int keysState,
+ int iter,
+ IgniteInternalFuture<?>...futs
+ ) throws IgniteInterruptedCheckedException {
+ assertTrue(waitForCondition(() -> {
+ for (int i = 0; i < keysCnt; i++) {
+ if (keysState == BOTH_EXISTS) {
+ if (!src.containsKey(i) || !dest.containsKey(i))
+ return checkFuts(false, futs);
+ }
+ else if (keysState == BOTH_REMOVED) {
+ if (src.containsKey(i) || dest.containsKey(i))
+ return checkFuts(false, futs);
+
+ continue;
+ }
+ else if (keysState == ANY_SAME_STATE) {
+ if (src.containsKey(i) != dest.containsKey(i))
+ return checkFuts(false, futs);
+
+ if (!src.containsKey(i))
+ continue;
+ }
+ else
+ throw new IllegalArgumentException(keysState + " not supported.");
+
+ Data data = dest.get(i);
+
+ if (!data.equals(src.get(i)))
+ return checkFuts(false, futs);
+
+ checkCRC(data, iter);
+ }
+
+ return checkFuts(true, futs);
+ }, getTestTimeout()));
+ }
+
+ /** */
+ private boolean checkFuts(boolean res, IgniteInternalFuture<?>...futs) {
+ for (IgniteInternalFuture<?> fut : futs)
+ assertFalse(fut.isDone());
+
+ return res;
+ }
+
+ /** */
+ public static void checkCRC(Data data, int iter) {
+ assertEquals(iter, data.iter);
+
+ crc.get().reset();
+ crc.get().update(ByteBuffer.wrap(data.payload), data.payload.length);
+
+ assertEquals(crc.get().getValue(), data.crc);
+ }
+
+ /**
+ * @param ign Ignite instance to watch for.
+ * @param topic Kafka topic name.
+ * @param caches Caches names to stream to kafka.
+ * @return Future for Change Data Capture application.
+ */
+ private IgniteInternalFuture<?> igniteToKafka(IgniteEx ign, String topic, String...caches) {
Review comment:
String...caches feature newer used.
So, seems we able to get gid of `new HashSet<>(Arrays.asList(caches))`
anyway, whitespace missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r650007796
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerConfiguration.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+
+/**
+ * Configuration of {@link KafkaToIgniteCdcStreamer} application.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see KafkaToIgniteLoader
+ */
+public class KafkaToIgniteCdcStreamerConfiguration {
+ /** Default {@link #kafkaParts} value. */
+ public static final int DFLT_PARTS = 16;
+
+ /** Default {@link #topic} value. */
+ public static final String DFLT_TOPIC = "ignite";
+
+ /** Default {@link #maxBatchSize} value. */
+ public static final int DFLT_MAX_BATCH_SIZE = 1024;
+
+ /** {@link Applier} thread count. */
+ private int threadCnt = DFLT_PARTS;
+
+ /** Topic name. */
+ private String topic = DFLT_TOPIC;
+
+ /** Kafka partitions count. */
+ private int kafkaParts = DFLT_PARTS;
Review comment:
> Do we really need 2 params with the same defaults?
Yes.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerConfiguration.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+
+/**
+ * Configuration of {@link KafkaToIgniteCdcStreamer} application.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see KafkaToIgniteLoader
+ */
+public class KafkaToIgniteCdcStreamerConfiguration {
+ /** Default {@link #kafkaParts} value. */
+ public static final int DFLT_PARTS = 16;
+
+ /** Default {@link #topic} value. */
+ public static final String DFLT_TOPIC = "ignite";
+
+ /** Default {@link #maxBatchSize} value. */
+ public static final int DFLT_MAX_BATCH_SIZE = 1024;
+
+ /** {@link Applier} thread count. */
+ private int threadCnt = DFLT_PARTS;
+
+ /** Topic name. */
+ private String topic = DFLT_TOPIC;
+
+ /** Kafka partitions count. */
+ private int kafkaParts = DFLT_PARTS;
Review comment:
> Seems that having kafkaParts is enough and some topics per thread may be required, instead of partPerApplier calculation?
Different servers have a different cores available for different loads.
I don't think we should do some implicit calculation here - the user knows better how many threads he wants to create for Streamer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655977288
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.CdcEventsApplier;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier extends CdcEventsApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
+
+ /** Caches ids to read. */
+ private final Set<Integer> caches;
+
+ /** Consumers. */
+ private final List<KafkaConsumer<Integer, byte[]>> cnsmrs = new ArrayList<>();
+
+ /** */
+ private final AtomicLong rcvdEvts = new AtomicLong();
+
+ /**
+ * @param ign Ignite instance.
+ * @param log Logger.
+ * @param kafkaProps Kafka properties.
+ * @param topic Topic name.
+ * @param kafkaPartFrom Read from partition.
+ * @param kafkaPartTo Read to partition.
+ * @param caches Cache ids.
+ * @param maxBatchSize Maximum batch size.
+ * @param closed Closed flag.
+ */
+ public KafkaToIgniteCdcStreamerApplier(
+ IgniteEx ign,
+ IgniteLogger log,
+ Properties kafkaProps,
+ String topic,
+ int kafkaPartFrom,
+ int kafkaPartTo,
+ Set<Integer> caches,
+ int maxBatchSize,
+ AtomicBoolean closed
+ ) {
+ super(maxBatchSize);
+
+ this.ign = ign;
+ this.kafkaProps = kafkaProps;
+ this.topic = topic;
+ this.kafkaPartFrom = kafkaPartFrom;
+ this.kafkaPartTo = kafkaPartTo;
+ this.caches = caches;
+ this.closed = closed;
+ this.log = log.getLogger(KafkaToIgniteCdcStreamerApplier.class);
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ U.setCurrentIgniteName(ign.name());
+
+ try {
+ for (int kafkaPart = kafkaPartFrom; kafkaPart < kafkaPartTo; kafkaPart++) {
+ KafkaConsumer<Integer, byte[]> cnsmr = new KafkaConsumer<>(kafkaProps);
+
+ cnsmr.assign(Collections.singleton(new TopicPartition(topic, kafkaPart)));
+
+ cnsmrs.add(cnsmr);
+ }
+
+ Iterator<KafkaConsumer<Integer, byte[]>> cnsmrIter = Collections.emptyIterator();
+
+ while (!closed.get()) {
+ if (!cnsmrIter.hasNext())
+ cnsmrIter = cnsmrs.iterator();
+
+ poll(cnsmrIter.next());
+ }
+ }
+ catch (WakeupException e) {
+ if (!closed.get())
+ log.error("Applier wakeup error!", e);
+ }
+ catch (Throwable e) {
+ log.error("Applier error!", e);
+
+ closed.set(true);
Review comment:
You're performing wake-up at another place when setting closed to true, why not here?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
+ closed.set(true);
+
+ appliers.forEach(U::closeQuiet);
Review comment:
you performing `closed.set(true);` twice
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean stopped = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ stopped
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
+ stopped.set(true);
+
+ appliers.forEach(U::closeQuiet);
+
+ log.info("Kafka to Ignite streamer interrupted");
Review comment:
always log an exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655277728
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Caches. */
+ private final Map<Integer, IgniteInternalCache<BinaryObject, BinaryObject>> ignCaches = new HashMap<>();
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Maximum batch size. */
+ private final int maxBatchSize;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
Review comment:
It's a common way to deal with bounaries.
Please, take a look at other standard JDK methods:
```
/**
beginIndex – the beginning index, inclusive.
endIndex – the ending index, exclusive.
*/
String#substring(int, int);
/**
beginIndex – the begin index, inclusive.
endIndex – the end index, exclusive.
*/
String#subSequence(int, int);
/**
fromIndex – low endpoint (inclusive) of the subList
toIndex – high endpoint (exclusive) of the subList
*/
List#subList(int, int);
```
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Caches. */
+ private final Map<Integer, IgniteInternalCache<BinaryObject, BinaryObject>> ignCaches = new HashMap<>();
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Maximum batch size. */
+ private final int maxBatchSize;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
Review comment:
It's a common way to deal with boundaries.
Please, take a look at other standard JDK methods:
```
/**
beginIndex – the beginning index, inclusive.
endIndex – the ending index, exclusive.
*/
String#substring(int, int);
/**
beginIndex – the begin index, inclusive.
endIndex – the end index, exclusive.
*/
String#subSequence(int, int);
/**
fromIndex – low endpoint (inclusive) of the subList
toIndex – high endpoint (exclusive) of the subList
*/
List#subList(int, int);
```
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Executor service to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final ExecutorService execSvc;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+
+ execSvc = Executors.newFixedThreadPool(streamerCfg.getThreadCount(), new ThreadFactory() {
+ private final AtomicInteger cntr = new AtomicInteger();
+
+ @Override public Thread newThread(@NotNull Runnable r) {
+ Thread th = new Thread(r);
+
+ th.setName("applier-thread-" + cntr.getAndIncrement());
+
+ return th;
+ }
+ });
Review comment:
Because we have convenient `ExecutorService`.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
Review comment:
Reason for what?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
+ closed.set(true);
+
+ appliers.forEach(U::closeQuiet);
Review comment:
`close` invocation required to call `KafkaConsumer#wakeUp` to cancel current consumer poll request.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.CdcEventsApplier;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier extends CdcEventsApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
+
+ /** Caches ids to read. */
+ private final Set<Integer> caches;
+
+ /** Consumers. */
+ private final List<KafkaConsumer<Integer, byte[]>> cnsmrs = new ArrayList<>();
+
+ /** */
+ private final AtomicLong rcvdEvts = new AtomicLong();
+
+ /**
+ * @param ign Ignite instance.
+ * @param log Logger.
+ * @param kafkaProps Kafka properties.
+ * @param topic Topic name.
+ * @param kafkaPartFrom Read from partition.
+ * @param kafkaPartTo Read to partition.
+ * @param caches Cache ids.
+ * @param maxBatchSize Maximum batch size.
+ * @param closed Closed flag.
+ */
+ public KafkaToIgniteCdcStreamerApplier(
+ IgniteEx ign,
+ IgniteLogger log,
+ Properties kafkaProps,
+ String topic,
+ int kafkaPartFrom,
+ int kafkaPartTo,
+ Set<Integer> caches,
+ int maxBatchSize,
+ AtomicBoolean closed
+ ) {
+ super(maxBatchSize);
+
+ this.ign = ign;
+ this.kafkaProps = kafkaProps;
+ this.topic = topic;
+ this.kafkaPartFrom = kafkaPartFrom;
+ this.kafkaPartTo = kafkaPartTo;
+ this.caches = caches;
+ this.closed = closed;
+ this.log = log.getLogger(KafkaToIgniteCdcStreamerApplier.class);
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ U.setCurrentIgniteName(ign.name());
+
+ try {
+ for (int kafkaPart = kafkaPartFrom; kafkaPart < kafkaPartTo; kafkaPart++) {
+ KafkaConsumer<Integer, byte[]> cnsmr = new KafkaConsumer<>(kafkaProps);
+
+ cnsmr.assign(Collections.singleton(new TopicPartition(topic, kafkaPart)));
+
+ cnsmrs.add(cnsmr);
+ }
+
+ Iterator<KafkaConsumer<Integer, byte[]>> cnsmrIter = Collections.emptyIterator();
+
+ while (!closed.get()) {
+ if (!cnsmrIter.hasNext())
+ cnsmrIter = cnsmrs.iterator();
+
+ poll(cnsmrIter.next());
+ }
+ }
+ catch (WakeupException e) {
+ if (!closed.get())
+ log.error("Applier wakeup error!", e);
+ }
+ catch (Throwable e) {
+ log.error("Applier error!", e);
+
+ closed.set(true);
Review comment:
I don't think so. Why?
All other appliers will try to perform last poll and will finish after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656214059
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = new IgniteEx[] {
+ startGrid(1),
+ startGrid(2)
+ };
+
+ srcClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ srcClusterCliCfg[i] = optimize(getConfiguration("src-cluster-client" + i).setClientMode(true));
+
+ srcCluster[0].cluster().state(ACTIVE);
+ srcCluster[0].cluster().tag("source");
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ commPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = new IgniteEx[] {
+ startGrid(4),
+ startGrid(5)
+ };
+
+ destClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ destClusterCliCfg[i] = optimize(getConfiguration("dest-cluster-client" + i).setClientMode(true));
+
+ assertFalse("source".equals(destCluster[0].cluster().tag()));
+
+ destCluster[0].cluster().state(ACTIVE);
+ destCluster[0].cluster().tag("destination");
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ List<IgniteInternalFuture<?>> futs = startActivePassiveCdc();
+
+ try {
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster[0].createCache(AP_CACHE);
+
+ destCache.put(1, ConflictResolvableTestData.create());
+ destCache.remove(1);
+
+ // Updates for "ignored-cache" should be ignored because of CDC consume configuration.
+ runAsync(generateData("ignored-cache", srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT)));
+ runAsync(generateData(AP_CACHE, srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT)));
+
+ List<IgniteInternalFuture<?>> k2iFut = startActivePassiveReplication();
+
+ if (k2iFut != null)
+ futs.addAll(k2iFut);
+
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster[srcCluster.length - 1].getOrCreateCache(AP_CACHE);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, EXISTS, futs);
+
+ IntStream.range(0, KEYS_CNT).forEach(srcCache::remove);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ @Test
+ public void testActiveActiveReplication() throws Exception {
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, srcCluster[srcCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 == 0)));
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, destCluster[destCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 != 0)));
+
+ List<IgniteInternalFuture<?>> futs = startActiveActiveCdc();
+
+ try {
+ List<IgniteInternalFuture<?>> replicationFuts = startActiveActiveReplication();
+
+ if (replicationFuts != null)
+ futs.addAll(replicationFuts);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, EXISTS, futs);
+
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 == 0).forEach(srcCache::remove));
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 != 0).forEach(destCache::remove));
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ public static Runnable generateData(String cacheName, IgniteEx ign, IntStream keys) {
+ return () -> {
+ IgniteCache<Integer, ConflictResolvableTestData> cache = ign.getOrCreateCache(cacheName);
+
+ keys.forEach(i -> cache.put(i, ConflictResolvableTestData.create()));
+ };
+ }
+
+ /** */
+ public void waitForSameData(
+ IgniteCache<Integer, ConflictResolvableTestData> src,
+ IgniteCache<Integer, ConflictResolvableTestData> dest,
+ int keysCnt,
+ int keysState,
+ List<IgniteInternalFuture<?>> futs
+ ) throws IgniteInterruptedCheckedException {
+ assertTrue(waitForCondition(() -> {
+ for (int i = 0; i < keysCnt; i++) {
+ if (keysState == EXISTS) {
+ if (!src.containsKey(i) || !dest.containsKey(i))
+ return checkFuts(false, futs);
+ }
+ else if (keysState == REMOVED) {
+ if (src.containsKey(i) || dest.containsKey(i))
+ return checkFuts(false, futs);
+
+ continue;
+ }
+ else
+ throw new IllegalArgumentException(keysState + " not supported.");
+
+ ConflictResolvableTestData data = dest.get(i);
+
+ if (!data.equals(src.get(i)))
+ return checkFuts(false, futs);
+ }
+
+ return checkFuts(true, futs);
+ }, getTestTimeout()));
+ }
+
+ /** */
+ private boolean checkFuts(boolean res, List<IgniteInternalFuture<?>> futs) {
+ for (int i = 0; i < futs.size(); i++)
+ assertFalse("Fut " + i, futs.get(i).isDone());
+
+ for (IgniteInternalFuture<?> fut : futs)
+ assertFalse(fut.isDone());
+
+ return res;
+ }
+
+ /** */
+ protected abstract List<IgniteInternalFuture<?>> startActivePassiveCdc();
Review comment:
ActivePassive/ActiveActive are common naming for cluster replication modes.
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = new IgniteEx[] {
+ startGrid(1),
+ startGrid(2)
+ };
+
+ srcClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ srcClusterCliCfg[i] = optimize(getConfiguration("src-cluster-client" + i).setClientMode(true));
+
+ srcCluster[0].cluster().state(ACTIVE);
+ srcCluster[0].cluster().tag("source");
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ commPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = new IgniteEx[] {
+ startGrid(4),
+ startGrid(5)
+ };
+
+ destClusterCliCfg = new IgniteConfiguration[clientsCnt];
+
+ for (int i = 0; i < clientsCnt; i++)
+ destClusterCliCfg[i] = optimize(getConfiguration("dest-cluster-client" + i).setClientMode(true));
+
+ assertFalse("source".equals(destCluster[0].cluster().tag()));
+
+ destCluster[0].cluster().state(ACTIVE);
+ destCluster[0].cluster().tag("destination");
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ List<IgniteInternalFuture<?>> futs = startActivePassiveCdc();
+
+ try {
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster[0].createCache(AP_CACHE);
+
+ destCache.put(1, ConflictResolvableTestData.create());
+ destCache.remove(1);
+
+ // Updates for "ignored-cache" should be ignored because of CDC consume configuration.
+ runAsync(generateData("ignored-cache", srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT)));
+ runAsync(generateData(AP_CACHE, srcCluster[srcCluster.length - 1], IntStream.range(0, KEYS_CNT)));
+
+ List<IgniteInternalFuture<?>> k2iFut = startActivePassiveReplication();
+
+ if (k2iFut != null)
+ futs.addAll(k2iFut);
+
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster[srcCluster.length - 1].getOrCreateCache(AP_CACHE);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, EXISTS, futs);
+
+ IntStream.range(0, KEYS_CNT).forEach(srcCache::remove);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ @Test
+ public void testActiveActiveReplication() throws Exception {
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, srcCluster[srcCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 == 0)));
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, destCluster[destCluster.length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 != 0)));
+
+ List<IgniteInternalFuture<?>> futs = startActiveActiveCdc();
+
+ try {
+ List<IgniteInternalFuture<?>> replicationFuts = startActiveActiveReplication();
+
+ if (replicationFuts != null)
+ futs.addAll(replicationFuts);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, EXISTS, futs);
+
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 == 0).forEach(srcCache::remove));
+ runAsync(() -> IntStream.range(0, KEYS_CNT).filter(j -> j % 2 != 0).forEach(destCache::remove));
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** */
+ public static Runnable generateData(String cacheName, IgniteEx ign, IntStream keys) {
+ return () -> {
+ IgniteCache<Integer, ConflictResolvableTestData> cache = ign.getOrCreateCache(cacheName);
+
+ keys.forEach(i -> cache.put(i, ConflictResolvableTestData.create()));
+ };
+ }
+
+ /** */
+ public void waitForSameData(
+ IgniteCache<Integer, ConflictResolvableTestData> src,
+ IgniteCache<Integer, ConflictResolvableTestData> dest,
+ int keysCnt,
+ int keysState,
+ List<IgniteInternalFuture<?>> futs
+ ) throws IgniteInterruptedCheckedException {
+ assertTrue(waitForCondition(() -> {
+ for (int i = 0; i < keysCnt; i++) {
+ if (keysState == EXISTS) {
+ if (!src.containsKey(i) || !dest.containsKey(i))
+ return checkFuts(false, futs);
+ }
+ else if (keysState == REMOVED) {
+ if (src.containsKey(i) || dest.containsKey(i))
+ return checkFuts(false, futs);
+
+ continue;
+ }
+ else
+ throw new IllegalArgumentException(keysState + " not supported.");
+
+ ConflictResolvableTestData data = dest.get(i);
+
+ if (!data.equals(src.get(i)))
+ return checkFuts(false, futs);
+ }
+
+ return checkFuts(true, futs);
+ }, getTestTimeout()));
+ }
+
+ /** */
+ private boolean checkFuts(boolean res, List<IgniteInternalFuture<?>> futs) {
+ for (int i = 0; i < futs.size(); i++)
+ assertFalse("Fut " + i, futs.get(i).isDone());
+
+ for (IgniteInternalFuture<?> fut : futs)
+ assertFalse(fut.isDone());
+
+ return res;
+ }
+
+ /** */
+ protected abstract List<IgniteInternalFuture<?>> startActivePassiveCdc();
+
+ /** */
+ protected abstract List<IgniteInternalFuture<?>> startActiveActiveCdc();
Review comment:
ActivePassive/ActiveActive are common naming for cluster replication modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648061089
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureKafkaToIgnite.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.cdc.conflictplugin.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link ChangeDataCaptureIgniteToKafka} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * Properties list:
+ * <ul>
+ * <li>{@link #KAFKA_TO_IGNITE_THREAD_COUNT} - count of {@link Applier} threads.</li>
+ * <li>{@link #IGNITE_TO_KAFKA_TOPIC} - Kafka topic name if not provided in constructor.</li>
+ * </ul>
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class ChangeDataCaptureKafkaToIgnite implements Runnable {
Review comment:
The name looks odd to me.
What role it has, what it do?
It's a sink or producer if we having a deal with patterrns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656165798
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
Review comment:
What's wrong with these numbers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r649769612
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/conflictplugin/CacheVersionConflictResolverCachePluginProvider.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictplugin;
+
+import javax.cache.Cache;
+import org.apache.ignite.cluster.ClusterNode;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.internal.processors.cache.CacheConflictResolutionManager;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.plugin.CachePluginConfiguration;
+import org.apache.ignite.plugin.CachePluginProvider;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Intermediate component to provide {@link CacheConflictResolutionManagerImpl} for specific cache.
+ *
+ * @see CacheConflictResolutionManagerImpl
+ * @see CacheVersionConflictResolverImpl
+ * @see CacheVersionConflictResolver
+ */
+public class CacheVersionConflictResolverCachePluginProvider<K, V, C extends CachePluginConfiguration<K, V>>
+ implements CachePluginProvider<C> {
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * Note, values of this field must implement {@link Comparable}.
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
Review comment:
Please mention this used only for external data applying.
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/conflictplugin/CacheVersionConflictResolverPluginProvider.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictplugin;
+
+import java.io.Serializable;
+import java.util.Set;
+import java.util.UUID;
+import org.apache.ignite.cluster.ClusterNode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.plugin.CachePluginContext;
+import org.apache.ignite.plugin.CachePluginProvider;
+import org.apache.ignite.plugin.ExtensionRegistry;
+import org.apache.ignite.plugin.IgnitePlugin;
+import org.apache.ignite.plugin.PluginConfiguration;
+import org.apache.ignite.plugin.PluginContext;
+import org.apache.ignite.plugin.PluginProvider;
+import org.apache.ignite.plugin.PluginValidationException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Plugin to enable {@link CacheVersionConflictResolverImpl} for provided caches.
+ *
+ * @see CacheVersionConflictResolverImpl
+ * @see CacheVersionConflictResolver
+ */
+public class CacheVersionConflictResolverPluginProvider<C extends PluginConfiguration> implements PluginProvider<C> {
+ /** Plugin context. */
+ private PluginContext ctx;
+
+ /** Cluster id. */
+ private byte clusterId;
+
+ /** Cache names. */
+ private Set<String> caches;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * Note, values of this field must implement {@link Comparable}.
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private String conflictResolveField;
+
+ /** Cache plugin provider. */
+ private CachePluginProvider<?> provider;
+
+ /** */
+ public CacheVersionConflictResolverPluginProvider() {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override public String name() {
+ return "cache-version-conflict-resolver";
Review comment:
How about the plain name like "Cache version ..."?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/conflictplugin/CacheVersionConflictResolverPluginProvider.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictplugin;
+
+import java.io.Serializable;
+import java.util.Set;
+import java.util.UUID;
+import org.apache.ignite.cluster.ClusterNode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.plugin.CachePluginContext;
+import org.apache.ignite.plugin.CachePluginProvider;
+import org.apache.ignite.plugin.ExtensionRegistry;
+import org.apache.ignite.plugin.IgnitePlugin;
+import org.apache.ignite.plugin.PluginConfiguration;
+import org.apache.ignite.plugin.PluginContext;
+import org.apache.ignite.plugin.PluginProvider;
+import org.apache.ignite.plugin.PluginValidationException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Plugin to enable {@link CacheVersionConflictResolverImpl} for provided caches.
+ *
+ * @see CacheVersionConflictResolverImpl
+ * @see CacheVersionConflictResolver
+ */
+public class CacheVersionConflictResolverPluginProvider<C extends PluginConfiguration> implements PluginProvider<C> {
+ /** Plugin context. */
+ private PluginContext ctx;
+
+ /** Cluster id. */
+ private byte clusterId;
+
+ /** Cache names. */
+ private Set<String> caches;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * Note, values of this field must implement {@link Comparable}.
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private String conflictResolveField;
+
+ /** Cache plugin provider. */
+ private CachePluginProvider<?> provider;
+
+ /** */
+ public CacheVersionConflictResolverPluginProvider() {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override public String name() {
+ return "cache-version-conflict-resolver";
+ }
+
+ /** {@inheritDoc} */
+ @Override public String version() {
+ return "0.0.1-SNAPSHOT";
Review comment:
1.0.0-SNAPSHOT
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
Review comment:
> conflicts(in
whitespace missed
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Executor service to run {@link Applier} instances. */
+ private final ExecutorService execSvc;
+
+ /** Appliers. */
+ private final List<Applier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+
+ execSvc = Executors.newFixedThreadPool(streamerCfg.getThreadCount(), new ThreadFactory() {
+ private final AtomicInteger cntr = new AtomicInteger();
+
+ @Override public Thread newThread(@NotNull Runnable r) {
+ Thread th = new Thread(r);
+
+ th.setName("applier-thread-" + cntr.getAndIncrement());
+
+ return th;
+ }
+ });
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(
+ igniteCfg.getGridLogger(),
+ "kafka-ignite-streamer",
+ igniteCfg.getNodeId() != null ? igniteCfg.getNodeId() : UUID.randomUUID(),
+ igniteCfg.getWorkDirectory()
+ );
Review comment:
duplicates code at ChangeDataCapture#ChangeDataCapture
Could we deduplicate this?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Executor service to run {@link Applier} instances. */
+ private final ExecutorService execSvc;
+
+ /** Appliers. */
+ private final List<Applier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+
+ execSvc = Executors.newFixedThreadPool(streamerCfg.getThreadCount(), new ThreadFactory() {
+ private final AtomicInteger cntr = new AtomicInteger();
+
+ @Override public Thread newThread(@NotNull Runnable r) {
+ Thread th = new Thread(r);
+
+ th.setName("applier-thread-" + cntr.getAndIncrement());
+
+ return th;
+ }
+ });
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(
+ igniteCfg.getGridLogger(),
+ "kafka-ignite-streamer",
+ igniteCfg.getNodeId() != null ? igniteCfg.getNodeId() : UUID.randomUUID(),
+ igniteCfg.getWorkDirectory()
+ );
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCacheNames())) {
+ caches = streamerCfg.getCacheNames().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int partPerApplier = streamerCfg.getKafkaPartitions() / streamerCfg.getThreadCount();
Review comment:
> streamerCfg.getKafkaPartitions() and streamerCfg.getThreadCount()
call, set to variable, use twice or more.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Executor service to run {@link Applier} instances. */
+ private final ExecutorService execSvc;
+
+ /** Appliers. */
+ private final List<Applier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
Review comment:
> streamerCfg.getThreadCount()
call, set to variable, use twice or more.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerConfiguration.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+
+/**
+ * Configuration of {@link KafkaToIgniteCdcStreamer} application.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see KafkaToIgniteLoader
+ */
+public class KafkaToIgniteCdcStreamerConfiguration {
+ /** Default {@link #kafkaParts} value. */
+ public static final int DFLT_PARTS = 16;
+
+ /** Default {@link #topic} value. */
+ public static final String DFLT_TOPIC = "ignite";
+
+ /** Default {@link #maxBatchSize} value. */
+ public static final int DFLT_MAX_BATCH_SIZE = 1024;
+
+ /** {@link Applier} thread count. */
+ private int threadCnt = DFLT_PARTS;
+
+ /** Topic name. */
+ private String topic = DFLT_TOPIC;
+
+ /** Kafka partitions count. */
+ private int kafkaParts = DFLT_PARTS;
Review comment:
Do we really need 2 params with the same defaults?
Seems that having `kafkaParts` is enough and some `topics per thread` may be required, instead of `partPerApplier` calculation?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Executor service to run {@link Applier} instances. */
+ private final ExecutorService execSvc;
+
+ /** Appliers. */
+ private final List<Applier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+
+ execSvc = Executors.newFixedThreadPool(streamerCfg.getThreadCount(), new ThreadFactory() {
+ private final AtomicInteger cntr = new AtomicInteger();
+
+ @Override public Thread newThread(@NotNull Runnable r) {
+ Thread th = new Thread(r);
+
+ th.setName("applier-thread-" + cntr.getAndIncrement());
+
+ return th;
+ }
+ });
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(
+ igniteCfg.getGridLogger(),
+ "kafka-ignite-streamer",
+ igniteCfg.getNodeId() != null ? igniteCfg.getNodeId() : UUID.randomUUID(),
+ igniteCfg.getWorkDirectory()
+ );
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCacheNames())) {
+ caches = streamerCfg.getCacheNames().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int partPerApplier = streamerCfg.getKafkaPartitions() / streamerCfg.getThreadCount();
+
+ for (int i = 0; i < streamerCfg.getThreadCount(); i++) {
+ int from = i * partPerApplier;
+
+ int to = (i == streamerCfg.getThreadCount() - 1)
+ ? streamerCfg.getKafkaPartitions() + 1
+ : (i + 1) * partPerApplier;
Review comment:
some black magic detected here :)
spent 2 minutes and still have no clue what's going on here %)
why not just iterate over the appliers and partitions, and append parts to appliers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r650724395
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerConfiguration.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+
+/**
+ * Configuration of {@link KafkaToIgniteCdcStreamer} application.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see KafkaToIgniteLoader
+ */
+public class KafkaToIgniteCdcStreamerConfiguration {
+ /** Default {@link #kafkaParts} value. */
+ public static final int DFLT_PARTS = 16;
+
+ /** Default {@link #topic} value. */
+ public static final String DFLT_TOPIC = "ignite";
+
+ /** Default {@link #maxBatchSize} value. */
+ public static final int DFLT_MAX_BATCH_SIZE = 1024;
+
+ /** {@link Applier} thread count. */
+ private int threadCnt = DFLT_PARTS;
+
+ /** Topic name. */
+ private String topic = DFLT_TOPIC;
+
+ /** Kafka partitions count. */
+ private int kafkaParts = DFLT_PARTS;
Review comment:
The current logic is not clear to me.
Having `threadCnt` and `kafkaParts` simultaneously with subsequent calculation `partPerApplier = kafkaParts / threadCnt` may set N partitions to some Appliers, but N+1 to others, that's not what user may expect.
We must simplify this to `kafkaParts` and `partPerApplier`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652451655
##########
File path: modules/cdc-ext/modules/core/src/test/config/tests.properties
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
Review comment:
Please get rid of this in your PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648048757
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
Review comment:
I like the current naming:
`IgniteToKafka` streams changes from Ignite to Kafka.
`KafkaToIgnite` streams changes from Kafka to Ignite.
What's wrong with it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652473912
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ o = ((BinaryObject)oldVal).field(conflictResolveField);
+ n = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ o = U.field(oldVal, conflictResolveField);
+ n = U.field(newVal, conflictResolveField);
+ }
+
+ if (o == null || n == null)
+ return o == null;
+
+ return o.compareTo(n) < 0;
+ }
+ catch (Exception e) {
+ log.error("Error while resolving replication conflict with field '" +
+ conflictResolveField + "'.\nConflict resolve with the field disabled.", e);
+
+ conflictResolveFieldEnabled = false;
+ }
+ }
+ }
+
+ // Cluster with the lower ID have biggest priority(e.g. first cluster is main).
+ return newEntry.dataCenterId() < oldEntry.dataCenterId();
Review comment:
Value of this flag analyzed during `toString` invocation.
Please, take a look at `CacheObjectAdapter#toString` and delaration of the `val` field:
```
public abstract class CacheObjectAdapter ... {
@GridToStringInclude(sensitive = true) //Note sensitive is true.
@GridDirectTransient
protected Object val;
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656289885
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String ACTIVE_PASSIVE_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 1;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 2;
+
+ /** */
+ private enum WaitDataMode {
+ /** */
+ EXISTS,
+
+ /** */
+ REMOVED
+ }
+
+ /** */
+ public static final int KEYS_CNT = 1000;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> srcCluster;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> destCluster;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = setupCluster("source", "src-cluster-client", 0);
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = setupCluster("destination", "dest-cluster-client", 2);
+ }
+
+ /** */
+ private IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> setupCluster(String clusterTag, String clientPrefix, int idx) throws Exception {
+ IgniteEx[] cluster = new IgniteEx[] {
+ startGrid(idx + 1),
+ startGrid(idx + 2)
+ };
+
+ IgniteConfiguration[] clusterCliCfg = new IgniteConfiguration[2];
+
+ for (int i = 0; i < 2; i++)
+ clusterCliCfg[i] = optimize(getConfiguration(clientPrefix + i).setClientMode(true));
+
+ cluster[0].cluster().state(ACTIVE);
+ cluster[0].cluster().tag(clusterTag);
+
+ return F.t(cluster, clusterCliCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** Active/Passive mode means changes made only in one cluster. */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ List<IgniteInternalFuture<?>> futs = startActivePassiveCdc();
+
+ try {
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster.get1()[0].createCache(ACTIVE_PASSIVE_CACHE);
+
+ destCache.put(1, ConflictResolvableTestData.create());
+ destCache.remove(1);
+
+ // Updates for "ignored-cache" should be ignored because of CDC consume configuration.
+ runAsync(generateData("ignored-cache", srcCluster.get1()[srcCluster.get1().length - 1], IntStream.range(0, KEYS_CNT)));
+ runAsync(generateData(ACTIVE_PASSIVE_CACHE, srcCluster.get1()[srcCluster.get1().length - 1], IntStream.range(0, KEYS_CNT)));
+
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache =
+ srcCluster.get1()[srcCluster.get1().length - 1].getOrCreateCache(ACTIVE_PASSIVE_CACHE);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, WaitDataMode.EXISTS, futs);
+
+ IntStream.range(0, KEYS_CNT).forEach(srcCache::remove);
+
+ waitForSameData(srcCache, destCache, KEYS_CNT, WaitDataMode.REMOVED, futs);
+ }
+ finally {
+ for (IgniteInternalFuture<?> fut : futs)
+ fut.cancel();
+ }
+ }
+
+ /** Active/Active mode means changes made in both clusters. */
+ @Test
+ public void testActiveActiveReplication() throws Exception {
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache = srcCluster.get1()[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster.get1()[0].getOrCreateCache(ACTIVE_ACTIVE_CACHE);
+
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, srcCluster.get1()[srcCluster.get1().length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 == 0)));
+ runAsync(generateData(ACTIVE_ACTIVE_CACHE, destCluster.get1()[destCluster.get1().length - 1],
+ IntStream.range(0, KEYS_CNT).filter(i -> i % 2 != 0)));
Review comment:
it's extremely hard to see the difference between the lines, could you provide a hint that keys are disjoint?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String ACTIVE_PASSIVE_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 1;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 2;
+
+ /** */
+ private enum WaitDataMode {
+ /** */
+ EXISTS,
+
+ /** */
+ REMOVED
+ }
+
+ /** */
+ public static final int KEYS_CNT = 1000;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> srcCluster;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> destCluster;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = setupCluster("source", "src-cluster-client", 0);
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = setupCluster("destination", "dest-cluster-client", 2);
+ }
+
+ /** */
+ private IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> setupCluster(String clusterTag, String clientPrefix, int idx) throws Exception {
+ IgniteEx[] cluster = new IgniteEx[] {
+ startGrid(idx + 1),
+ startGrid(idx + 2)
+ };
+
+ IgniteConfiguration[] clusterCliCfg = new IgniteConfiguration[2];
+
+ for (int i = 0; i < 2; i++)
+ clusterCliCfg[i] = optimize(getConfiguration(clientPrefix + i).setClientMode(true));
+
+ cluster[0].cluster().state(ACTIVE);
+ cluster[0].cluster().tag(clusterTag);
Review comment:
any reason for setting the tag in case it never checked?
could we add the check that cluster1.tag!=cluster2.tag!=null?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/CdcKafkaReplicationTest.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import org.apache.ignite.cdc.AbstractReplicationTest;
+import org.apache.ignite.cdc.ChangeDataCaptureConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.junit.ClassRule;
+
+import static org.apache.ignite.cdc.kafka.KafkaToIgniteCdcStreamerConfiguration.DFLT_PARTS;
+import static org.apache.ignite.cdc.kafka.KafkaToIgniteCdcStreamerConfiguration.DFLT_TOPIC;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+
+/**
+ * Tests for kafka replication.
+ */
+public class CdcKafkaReplicationTest extends AbstractReplicationTest {
+ /** */
+ public static final String SRC_DEST_TOPIC = "source-dest";
+
+ /** */
+ public static final String DEST_SRC_TOPIC = "dest-source";
+
+ /** */
+ @ClassRule
+ public static final EmbeddedKafkaCluster KAFKA = new EmbeddedKafkaCluster(1);
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ super.beforeTest();
+
+ KAFKA.start();
+
+ KAFKA.createTopic(DFLT_TOPIC, DFLT_PARTS, 1);
+ KAFKA.createTopic(SRC_DEST_TOPIC, DFLT_PARTS, 1);
+ KAFKA.createTopic(DEST_SRC_TOPIC, DFLT_PARTS, 1);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ KAFKA.deleteAllTopicsAndWait(getTestTimeout());
+ }
+
+ /** {@inheritDoc} */
+ @Override protected List<IgniteInternalFuture<?>> startActivePassiveCdc() {
+ List<IgniteInternalFuture<?>> futs = new ArrayList<>();
+
+ for (int i = 0; i < srcCluster.get1().length; i++)
+ futs.add(igniteToKafka(srcCluster.get1()[i].configuration(), DFLT_TOPIC, ACTIVE_PASSIVE_CACHE));
+
+ for (int i = 0; i < destCluster.get1().length; i++) {
+ futs.add(kafkaToIgnite(
+ ACTIVE_PASSIVE_CACHE,
+ DFLT_TOPIC,
+ destCluster.get2()[i],
+ i * (DFLT_PARTS / 2),
+ (i + 1) * (DFLT_PARTS / 2)
+ ));
+ }
+
+
+ return futs;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected List<IgniteInternalFuture<?>> startActiveActiveCdc() {
+ List<IgniteInternalFuture<?>> futs = new ArrayList<>();
+
+ for (int i = 0; i < srcCluster.get1().length; i++)
+ futs.add(igniteToKafka(srcCluster.get1()[i].configuration(), SRC_DEST_TOPIC, AbstractReplicationTest.ACTIVE_ACTIVE_CACHE));
+
+ for (int i = 0; i < destCluster.get1().length; i++)
+ futs.add(igniteToKafka(destCluster.get1()[i].configuration(), DEST_SRC_TOPIC, AbstractReplicationTest.ACTIVE_ACTIVE_CACHE));
+
+ futs.add(kafkaToIgnite(ACTIVE_ACTIVE_CACHE, SRC_DEST_TOPIC, destCluster.get2()[0], 0, DFLT_PARTS));
+
Review comment:
no newline required
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String ACTIVE_PASSIVE_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 1;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 2;
+
+ /** */
+ private enum WaitDataMode {
+ /** */
+ EXISTS,
+
+ /** */
+ REMOVED
+ }
+
+ /** */
+ public static final int KEYS_CNT = 1000;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> srcCluster;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> destCluster;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = setupCluster("source", "src-cluster-client", 0);
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = setupCluster("destination", "dest-cluster-client", 2);
+ }
+
+ /** */
+ private IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> setupCluster(String clusterTag, String clientPrefix, int idx) throws Exception {
+ IgniteEx[] cluster = new IgniteEx[] {
+ startGrid(idx + 1),
+ startGrid(idx + 2)
+ };
+
+ IgniteConfiguration[] clusterCliCfg = new IgniteConfiguration[2];
+
+ for (int i = 0; i < 2; i++)
+ clusterCliCfg[i] = optimize(getConfiguration(clientPrefix + i).setClientMode(true));
+
+ cluster[0].cluster().state(ACTIVE);
+ cluster[0].cluster().tag(clusterTag);
+
+ return F.t(cluster, clusterCliCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** Active/Passive mode means changes made only in one cluster. */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ List<IgniteInternalFuture<?>> futs = startActivePassiveCdc();
+
+ try {
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster.get1()[0].createCache(ACTIVE_PASSIVE_CACHE);
+
+ destCache.put(1, ConflictResolvableTestData.create());
+ destCache.remove(1);
+
+ // Updates for "ignored-cache" should be ignored because of CDC consume configuration.
+ runAsync(generateData("ignored-cache", srcCluster.get1()[srcCluster.get1().length - 1], IntStream.range(0, KEYS_CNT)));
+ runAsync(generateData(ACTIVE_PASSIVE_CACHE, srcCluster.get1()[srcCluster.get1().length - 1], IntStream.range(0, KEYS_CNT)));
+
+ IgniteCache<Integer, ConflictResolvableTestData> srcCache =
+ srcCluster.get1()[srcCluster.get1().length - 1].getOrCreateCache(ACTIVE_PASSIVE_CACHE);
Review comment:
could we have `srcCluster.get1()` as a variable?
this should simplify multiply call to this
same for `testActiveActiveReplication()` including `destCluster.get1()`
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String ACTIVE_PASSIVE_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 1;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 2;
+
+ /** */
+ private enum WaitDataMode {
+ /** */
+ EXISTS,
+
+ /** */
+ REMOVED
+ }
+
+ /** */
+ public static final int KEYS_CNT = 1000;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> srcCluster;
+
+ /** */
+ protected static IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> destCluster;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}));
+
+ if (!cfg.isClientMode()) {
+ CacheVersionConflictResolverPluginProvider<?> cfgPlugin = new CacheVersionConflictResolverPluginProvider<>();
+
+ cfgPlugin.setClusterId(clusterId);
+ cfgPlugin.setCaches(new HashSet<>(Collections.singletonList(ACTIVE_ACTIVE_CACHE)));
+ cfgPlugin.setConflictResolveField("reqId");
+
+ cfg.setPluginProviders(cfgPlugin);
+
+ cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true)));
+
+ cfg.getDataStorageConfiguration()
+ .setWalForceArchiveTimeout(5_000)
+ .setChangeDataCaptureEnabled(true);
+
+ cfg.setConsistentId(igniteInstanceName);
+ }
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+
+ srcCluster = setupCluster("source", "src-cluster-client", 0);
+
+ discoPort += DFLT_PORT_RANGE + 1;
+ clusterId = DEST_CLUSTER_ID;
+
+ destCluster = setupCluster("destination", "dest-cluster-client", 2);
+ }
+
+ /** */
+ private IgniteBiTuple<IgniteEx[], IgniteConfiguration[]> setupCluster(String clusterTag, String clientPrefix, int idx) throws Exception {
+ IgniteEx[] cluster = new IgniteEx[] {
+ startGrid(idx + 1),
+ startGrid(idx + 2)
+ };
+
+ IgniteConfiguration[] clusterCliCfg = new IgniteConfiguration[2];
+
+ for (int i = 0; i < 2; i++)
+ clusterCliCfg[i] = optimize(getConfiguration(clientPrefix + i).setClientMode(true));
+
+ cluster[0].cluster().state(ACTIVE);
+ cluster[0].cluster().tag(clusterTag);
+
+ return F.t(cluster, clusterCliCfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** Active/Passive mode means changes made only in one cluster. */
+ @Test
+ public void testActivePassiveReplication() throws Exception {
+ List<IgniteInternalFuture<?>> futs = startActivePassiveCdc();
+
+ try {
+ IgniteCache<Integer, ConflictResolvableTestData> destCache = destCluster.get1()[0].createCache(ACTIVE_PASSIVE_CACHE);
+
+ destCache.put(1, ConflictResolvableTestData.create());
+ destCache.remove(1);
+
+ // Updates for "ignored-cache" should be ignored because of CDC consume configuration.
Review comment:
looks like you newer check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648052880
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/KafkaUtils.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.DescribeTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.common.errors.TopicExistsException;
+import org.apache.kafka.common.errors.UnknownTopicOrPartitionException;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/** Kafka Utils. */
+public class KafkaUtils {
+ /** Default kafka request timeout. */
+ public static final int TIMEOUT_MIN = 1;
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_NUM_PARTITIONS = "ignite.to.kafka.numpartitions";
Review comment:
Count means counted (incrementally).
Amount means size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656161658
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
Review comment:
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656168940
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
Review comment:
https://ignite.apache.org/docs/latest/clustering/tcp-ip-discovery#isolated-clusters-on-same-set-of-machines - I've followed the documentation to setup isolated clusters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656235203
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/CdcKafkaReplicationTest.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import org.apache.ignite.cdc.AbstractReplicationTest;
+import org.apache.ignite.cdc.ChangeDataCaptureConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.junit.ClassRule;
+
+import static org.apache.ignite.cdc.kafka.KafkaToIgniteCdcStreamerConfiguration.DFLT_PARTS;
+import static org.apache.ignite.cdc.kafka.KafkaToIgniteCdcStreamerConfiguration.DFLT_TOPIC;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+
+/**
+ * Tests for kafka replication.
+ */
+public class CdcKafkaReplicationTest extends AbstractReplicationTest {
+ /** */
+ public static final String SRC_DEST_TOPIC = "source-dest";
+
+ /** */
+ public static final String DEST_SRC_TOPIC = "dest-source";
+
+ /** */
+ protected static Properties props;
+
+ /** */
+ @ClassRule
+ public static final EmbeddedKafkaCluster KAFKA = new EmbeddedKafkaCluster(1);
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ super.beforeTest();
+
+ KAFKA.start();
+
+ if (props == null) {
+ props = new Properties();
+
+ props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA.bootstrapServers());
+ props.put(ConsumerConfig.GROUP_ID_CONFIG, "kafka-to-ignite-applier");
+ props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
+ props.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, "10000");
+ }
Review comment:
Do we need to perform this between `KAFKA.start();` and `KAFKA.createTopics...` ?
Seems this used for consumers only?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/kafka/CdcKafkaReplicationAppsTest.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.PrintWriter;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.startup.cmdline.ChangeDataCaptureCommandLineStartup;
+
+import static org.apache.ignite.cdc.kafka.KafkaToIgniteCdcStreamerConfiguration.DFLT_PARTS;
+import static org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+
+/** */
+public class CdcKafkaReplicationAppsTest extends CdcKafkaReplicationTest {
+ /** */
+ public static final String INSTANCE_NAME = "INSTANCE_NAME";
+
+ /** */
+ public static final String DISCO_PORT = "DISCO_PORT";
+
+ /** */
+ public static final String DISCO_PORT_RANGE = "DISCO_PORT_RANGE";
+
+ /** */
+ public static final String REPLICATED_CACHE = "REPLICATED_CACHE";
+
+ /** */
+ public static final String TOPIC = "TOPIC";
+
+ /** */
+ public static final String CONSISTENT_ID = "CONSISTENT_ID";
+
+ /** */
+ public static final String KAFKA_PARTS = "KAFKA_PARTS";
+
+ /** */
+ public static final String KAFKA_PARTS_FROM = "KAFKA_PARTS_FROM";
+
+ /** */
+ public static final String KAFKA_PARTS_TO = "KAFKA_PARTS_TO";
+
+ /** */
+ public static final String THREAD_CNT = "THREAD_CNT";
+
+ /** */
+ public static final String MAX_BATCH_SIZE = "MAX_BATCH_SIZE";
+
+ /** */
+ public static final String KAFKA_PROPS_PATH = "KAFKA_PROPS_PATH";
+
+ /** */
+ private String kafkaPropsPath = null;
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ super.beforeTest();
+
+ if (kafkaPropsPath == null) {
+ File file = File.createTempFile("kafka", "properties");
+
+ file.deleteOnExit();
+
+ try (FileOutputStream fos = new FileOutputStream(file)) {
+ props.store(fos, null);
+ }
+
+ kafkaPropsPath = "file://" + file.getAbsolutePath();
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override protected IgniteInternalFuture<?> igniteToKafka(IgniteConfiguration igniteCfg, String topic, String cache) {
+ Map<String, String> params = new HashMap<>();
+
+ params.put(INSTANCE_NAME, igniteCfg.getIgniteInstanceName());
+ params.put(REPLICATED_CACHE, cache);
+ params.put(TOPIC, topic);
+ params.put(CONSISTENT_ID, String.valueOf(igniteCfg.getConsistentId()));
+ params.put(KAFKA_PARTS, Integer.toString(DFLT_PARTS));
+ params.put(MAX_BATCH_SIZE, Integer.toString(KEYS_CNT));
+ params.put(KAFKA_PROPS_PATH, kafkaPropsPath);
+
+ return runAsync(
+ () -> ChangeDataCaptureCommandLineStartup.main(new String[] {prepareConfig("replication/ignite-to-kafka.xml", params)})
Review comment:
special namespaces for all Kafka configs seem to be useless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655285800
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Executor service to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final ExecutorService execSvc;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+
+ execSvc = Executors.newFixedThreadPool(streamerCfg.getThreadCount(), new ThreadFactory() {
+ private final AtomicInteger cntr = new AtomicInteger();
+
+ @Override public Thread newThread(@NotNull Runnable r) {
+ Thread th = new Thread(r);
+
+ th.setName("applier-thread-" + cntr.getAndIncrement());
+
+ return th;
+ }
+ });
Review comment:
Because we have convenient `ExecutorService`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r647398276
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/KafkaUtils.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.DescribeTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.common.errors.TopicExistsException;
+import org.apache.kafka.common.errors.UnknownTopicOrPartitionException;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/** Kafka Utils. */
+public class KafkaUtils {
+ /** Default kafka request timeout. */
+ public static final int TIMEOUT_MIN = 1;
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_NUM_PARTITIONS = "ignite.to.kafka.numpartitions";
Review comment:
partitions.amount?
BTW, proper place for this property is a `ChangeDataCaptureIgniteToKafka`
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/KafkaUtils.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.DescribeTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.common.errors.TopicExistsException;
+import org.apache.kafka.common.errors.UnknownTopicOrPartitionException;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/** Kafka Utils. */
+public class KafkaUtils {
+ /** Default kafka request timeout. */
+ public static final int TIMEOUT_MIN = 1;
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_NUM_PARTITIONS = "ignite.to.kafka.numpartitions";
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_REPLICATION_FACTOR = "ignite.to.kafka.replication.factor";
Review comment:
The proper place for this property is a `ChangeDataCaptureIgniteToKafka`
and this means you can get rid of IGNITE_TO_KAFKA` prefix
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureKafkaToIgnite.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.cdc.conflictplugin.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link ChangeDataCaptureIgniteToKafka} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * Properties list:
+ * <ul>
+ * <li>{@link #KAFKA_TO_IGNITE_THREAD_COUNT} - count of {@link Applier} threads.</li>
+ * <li>{@link #IGNITE_TO_KAFKA_TOPIC} - Kafka topic name if not provided in constructor.</li>
+ * </ul>
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class ChangeDataCaptureKafkaToIgnite implements Runnable {
Review comment:
KafkaChangeDataCaptureSink/Producer?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureKafkaToIgnite.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.cdc.conflictplugin.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link ChangeDataCaptureIgniteToKafka} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * Properties list:
+ * <ul>
+ * <li>{@link #KAFKA_TO_IGNITE_THREAD_COUNT} - count of {@link Applier} threads.</li>
+ * <li>{@link #IGNITE_TO_KAFKA_TOPIC} - Kafka topic name if not provided in constructor.</li>
+ * </ul>
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class ChangeDataCaptureKafkaToIgnite implements Runnable {
+ /** Property to define number of {@link Applier} threads. */
+ private static final String KAFKA_TO_IGNITE_THREAD_COUNT = "kafka.to.ignite.thread.count";
+
+ /** Ignite to Kafka topic name. */
+ public static final String IGNITE_TO_KAFKA_TOPIC = "ignite.to.kafka.topic";
+
+ /** Ignite to Kafka maximum batch size. */
+ public static final String IGNITE_TO_KAFKA_MAX_BATCH_SZ = "ignite.to.kafka.max.batch.size";
Review comment:
MAX_BATCH_SIZE
why ignite TO kafka?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/KafkaUtils.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.DescribeTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.common.errors.TopicExistsException;
+import org.apache.kafka.common.errors.UnknownTopicOrPartitionException;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/** Kafka Utils. */
+public class KafkaUtils {
+ /** Default kafka request timeout. */
+ public static final int TIMEOUT_MIN = 1;
Review comment:
DFLT_TIMEOUT?
resolution should be explained at Javadoc
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
+ /** Log. */
+ @LoggerResource
+ private IgniteLogger log;
+
+ /** Kafka producer to stream events. */
+ private KafkaProducer<Integer, ChangeDataCaptureEvent> producer;
+
+ /** Handle only primary entry flag. */
+ private final boolean onlyPrimary;
+
+ /** Topic name. */
+ private final String topic;
+
+ /** Number Kafka topic partitions. */
+ private int kafkaPartitionsNum;
+
+ /** Cache IDs. */
+ private final Set<Integer> cachesIds;
+
+ /** Max batch size. */
+ private final int maxBatchSz;
+
+ /** Kafka properties. */
+ private Properties kafkaProps;
+
+ /** Count of sent messages. */
+ private long cntSntMsgs;
Review comment:
msgCnt?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/Applier.java
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link ChangeDataCaptureIgniteToKafka} CDC consumer.
+ * <p>
+ * Each applier receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link ChangeDataCaptureKafkaToIgnite} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class Applier implements Runnable, AutoCloseable {
Review comment:
CdcApplier?
Class name meaning can be simplified by prefixes.
Let's call all the classes with prefixes.
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureKafkaToIgnite.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.cdc.conflictplugin.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link ChangeDataCaptureIgniteToKafka} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * Properties list:
+ * <ul>
+ * <li>{@link #KAFKA_TO_IGNITE_THREAD_COUNT} - count of {@link Applier} threads.</li>
+ * <li>{@link #IGNITE_TO_KAFKA_TOPIC} - Kafka topic name if not provided in constructor.</li>
+ * </ul>
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class ChangeDataCaptureKafkaToIgnite implements Runnable {
+ /** Property to define number of {@link Applier} threads. */
+ private static final String KAFKA_TO_IGNITE_THREAD_COUNT = "kafka.to.ignite.thread.count";
+
+ /** Ignite to Kafka topic name. */
+ public static final String IGNITE_TO_KAFKA_TOPIC = "ignite.to.kafka.topic";
Review comment:
KAFKA_TOPIC
why ignite TO kafka?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
+ /** Log. */
+ @LoggerResource
+ private IgniteLogger log;
+
+ /** Kafka producer to stream events. */
+ private KafkaProducer<Integer, ChangeDataCaptureEvent> producer;
+
+ /** Handle only primary entry flag. */
+ private final boolean onlyPrimary;
+
+ /** Topic name. */
+ private final String topic;
+
+ /** Number Kafka topic partitions. */
+ private int kafkaPartitionsNum;
+
+ /** Cache IDs. */
+ private final Set<Integer> cachesIds;
+
+ /** Max batch size. */
+ private final int maxBatchSz;
+
+ /** Kafka properties. */
+ private Properties kafkaProps;
+
+ /** Count of sent messages. */
+ private long cntSntMsgs;
+
+ /**
+ * @param topic Topic name.
+ * @param caches Cache names.
+ * @param maxBatchSz Maximum count of concurrently
+ * @param onlyPrimary If {@code true} then stream only events from primaries.
+ * @param kafkaProps Kafpa properties.
+ */
+ public ChangeDataCaptureIgniteToKafka(String topic, Set<String> caches, int maxBatchSz, boolean onlyPrimary, Properties kafkaProps) {
+ assert caches != null && !caches.isEmpty();
+
+ this.topic = topic;
+ this.onlyPrimary = onlyPrimary;
+ this.kafkaProps = kafkaProps;
+ this.maxBatchSz = maxBatchSz;
+
+ cachesIds = caches.stream()
+ .mapToInt(CU::cacheId)
+ .boxed()
+ .collect(Collectors.toSet());
+ }
+
+ /** {@inheritDoc} */
+ @Override public boolean onEvents(Iterator<ChangeDataCaptureEvent> evts) {
+ List<Future<RecordMetadata>> futs = new ArrayList<>();
+
+ while (evts.hasNext() && futs.size() <= maxBatchSz) {
+ ChangeDataCaptureEvent evt = evts.next();
+
+ if (onlyPrimary && !evt.primary())
+ continue;
+
+ if (evt.version().otherClusterVersion() != null)
+ continue;
+
+ if (!cachesIds.isEmpty() && !cachesIds.contains(evt.cacheId()))
+ continue;
+
+ cntSntMsgs++;
+
+ futs.add(producer.send(new ProducerRecord<>(
+ topic,
+ evt.partition() % kafkaPartitionsNum,
+ evt.cacheId(),
+ evt
+ )));
+ }
+
+ try {
+ for (Future<RecordMetadata> fut : futs)
+ fut.get(KafkaUtils.TIMEOUT_MIN, TimeUnit.MINUTES);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e) {
+ throw new RuntimeException(e);
+ }
+
+ if (log.isDebugEnabled())
+ log.debug("Events processed [cntSntMsgs=" + cntSntMsgs + ']');
+
+ return true;
+ }
+
+ /** {@inheritDoc} */
+ @Override public void start() {
+ try {
+ kafkaPartitionsNum = KafkaUtils.initTopic(topic, kafkaProps);
+
+ producer = new KafkaProducer<>(kafkaProps);
+
+ log.info("Ignite To Kafka started [topic=" + topic + ", onlyPrimary=" + onlyPrimary + ", cacheIds=" + cachesIds + ']');
Review comment:
Ignite To Kafka consumer/producer?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
Review comment:
KafkaChangeDataCaptureConsumer?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/KafkaUtils.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.DescribeTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.admin.TopicDescription;
+import org.apache.kafka.common.errors.TopicExistsException;
+import org.apache.kafka.common.errors.UnknownTopicOrPartitionException;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/** Kafka Utils. */
+public class KafkaUtils {
+ /** Default kafka request timeout. */
+ public static final int TIMEOUT_MIN = 1;
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_NUM_PARTITIONS = "ignite.to.kafka.numpartitions";
+
+ /** Ignite to Kafka topic partitions number. */
+ public static final String IGNITE_TO_KAFKA_REPLICATION_FACTOR = "ignite.to.kafka.replication.factor";
+
+ /**
+ * Initialize Kafka topic.
+ *
+ * @param props Properties.
+ */
+ public static int initTopic(String topic, Properties props)
+ throws InterruptedException, ExecutionException, TimeoutException {
+ try (AdminClient adminCli = AdminClient.create(props)) {
+ return createTopic(
+ topic,
+ Integer.parseInt(property(IGNITE_TO_KAFKA_NUM_PARTITIONS, props, "32")),
+ property(IGNITE_TO_KAFKA_REPLICATION_FACTOR, props, "1"),
Review comment:
let's make defaults constants
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureKafkaToIgnite.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.cdc.conflictplugin.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link ChangeDataCaptureIgniteToKafka} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * Properties list:
+ * <ul>
+ * <li>{@link #KAFKA_TO_IGNITE_THREAD_COUNT} - count of {@link Applier} threads.</li>
+ * <li>{@link #IGNITE_TO_KAFKA_TOPIC} - Kafka topic name if not provided in constructor.</li>
+ * </ul>
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class ChangeDataCaptureKafkaToIgnite implements Runnable {
+ /** Property to define number of {@link Applier} threads. */
+ private static final String KAFKA_TO_IGNITE_THREAD_COUNT = "kafka.to.ignite.thread.count";
Review comment:
THREAD_COUNT
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
+ /** Log. */
+ @LoggerResource
+ private IgniteLogger log;
+
+ /** Kafka producer to stream events. */
+ private KafkaProducer<Integer, ChangeDataCaptureEvent> producer;
+
+ /** Handle only primary entry flag. */
+ private final boolean onlyPrimary;
+
+ /** Topic name. */
+ private final String topic;
+
+ /** Number Kafka topic partitions. */
+ private int kafkaPartitionsNum;
+
+ /** Cache IDs. */
+ private final Set<Integer> cachesIds;
+
+ /** Max batch size. */
+ private final int maxBatchSz;
+
+ /** Kafka properties. */
+ private Properties kafkaProps;
+
+ /** Count of sent messages. */
+ private long cntSntMsgs;
+
+ /**
+ * @param topic Topic name.
+ * @param caches Cache names.
+ * @param maxBatchSz Maximum count of concurrently
+ * @param onlyPrimary If {@code true} then stream only events from primaries.
+ * @param kafkaProps Kafpa properties.
+ */
+ public ChangeDataCaptureIgniteToKafka(String topic, Set<String> caches, int maxBatchSz, boolean onlyPrimary, Properties kafkaProps) {
+ assert caches != null && !caches.isEmpty();
+
+ this.topic = topic;
+ this.onlyPrimary = onlyPrimary;
+ this.kafkaProps = kafkaProps;
+ this.maxBatchSz = maxBatchSz;
+
+ cachesIds = caches.stream()
+ .mapToInt(CU::cacheId)
+ .boxed()
+ .collect(Collectors.toSet());
+ }
+
+ /** {@inheritDoc} */
+ @Override public boolean onEvents(Iterator<ChangeDataCaptureEvent> evts) {
+ List<Future<RecordMetadata>> futs = new ArrayList<>();
+
+ while (evts.hasNext() && futs.size() <= maxBatchSz) {
+ ChangeDataCaptureEvent evt = evts.next();
+
+ if (onlyPrimary && !evt.primary())
+ continue;
+
+ if (evt.version().otherClusterVersion() != null)
+ continue;
+
+ if (!cachesIds.isEmpty() && !cachesIds.contains(evt.cacheId()))
+ continue;
+
+ cntSntMsgs++;
+
+ futs.add(producer.send(new ProducerRecord<>(
+ topic,
+ evt.partition() % kafkaPartitionsNum,
+ evt.cacheId(),
+ evt
+ )));
+ }
+
+ try {
+ for (Future<RecordMetadata> fut : futs)
+ fut.get(KafkaUtils.TIMEOUT_MIN, TimeUnit.MINUTES);
+ }
+ catch (InterruptedException | ExecutionException | TimeoutException e) {
+ throw new RuntimeException(e);
+ }
+
+ if (log.isDebugEnabled())
+ log.debug("Events processed [cntSntMsgs=" + cntSntMsgs + ']');
+
+ return true;
+ }
+
+ /** {@inheritDoc} */
+ @Override public void start() {
+ try {
+ kafkaPartitionsNum = KafkaUtils.initTopic(topic, kafkaProps);
+
+ producer = new KafkaProducer<>(kafkaProps);
+
+ log.info("Ignite To Kafka started [topic=" + topic + ", onlyPrimary=" + onlyPrimary + ", cacheIds=" + cachesIds + ']');
+ }
+ catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Sets kafka properties.
+ *
+ * @param kafkaProps Kafka properties.
+ */
+ public void setKafkaProps(Properties kafkaProps) {
+ this.kafkaProps = kafkaProps;
+ }
Review comment:
Could we have implementation methods located close to each other.
Let's relocate this method to the end of the file.
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
+ /** Log. */
+ @LoggerResource
+ private IgniteLogger log;
+
+ /** Kafka producer to stream events. */
+ private KafkaProducer<Integer, ChangeDataCaptureEvent> producer;
+
+ /** Handle only primary entry flag. */
+ private final boolean onlyPrimary;
+
+ /** Topic name. */
+ private final String topic;
+
+ /** Number Kafka topic partitions. */
+ private int kafkaPartitionsNum;
+
+ /** Cache IDs. */
+ private final Set<Integer> cachesIds;
+
+ /** Max batch size. */
+ private final int maxBatchSz;
Review comment:
maxBatchSize?
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
+ /** Log. */
+ @LoggerResource
+ private IgniteLogger log;
+
+ /** Kafka producer to stream events. */
+ private KafkaProducer<Integer, ChangeDataCaptureEvent> producer;
+
+ /** Handle only primary entry flag. */
+ private final boolean onlyPrimary;
+
+ /** Topic name. */
+ private final String topic;
+
+ /** Number Kafka topic partitions. */
+ private int kafkaPartitionsNum;
Review comment:
amount
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r651965049
##########
File path: modules/cdc-ext/modules/core/src/test/config/tests.properties
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
Review comment:
AFAICU these files exists in each extension module is just a copy of `tests.properties` from core module sources.
It seems to me, we should refactor to get rid of all of them.
But, this is out of the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655525474
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
Review comment:
Reason for what?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648061510
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
Review comment:
> What role it has, what it do?
`ChangeDataCaptureIgniteToKafka` - CDC consumer that stream change from Ignite to Kafka topic :)
> It's a KafkaConsumer because it's consuming to Kafka.
This class consumes FROM Ignite and produces TO Kafka :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652533484
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable oldResolveField;
+ Comparable newResolveField;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ oldResolveField = ((BinaryObject)oldVal).field(conflictResolveField);
+ newResolveField = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ oldResolveField = U.field(oldVal, conflictResolveField);
+ newResolveField = U.field(newVal, conflictResolveField);
+ }
+
+ if (oldResolveField == null || newResolveField == null)
+ return oldResolveField == null;
+
+ return oldResolveField.compareTo(newResolveField) < 0;
+ }
+ catch (Exception e) {
Review comment:
Not a cluster, but CDC.
I thought our strategy is to fail fast?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656001829
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
Review comment:
Could we have consistent naming?
Also, what are AA and AP? Could we explain this at Javadoc?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
Review comment:
Any reason for this default when we redefining this at `beforeTest()`?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
Review comment:
Looks like enum values
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
Review comment:
should this be a method with default implementation?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
Review comment:
magic numbers?
could we avoid this?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
Review comment:
The value is too small to find anything unusual.
How about thousands?
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
Review comment:
Could be reversed to have glued semantic units.
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
+
+ /** */
+ public static final int KEYS_CNT = 50;
+
+ /** */
+ protected static IgniteEx[] srcCluster;
+
+ /** */
+ protected static IgniteConfiguration[] srcClusterCliCfg;
+
+ /** */
+ protected static IgniteEx[] destCluster;
+
+ /** */
+ protected static IgniteConfiguration[] destClusterCliCfg;
+
+ /** */
+ private int commPort = TcpCommunicationSpi.DFLT_PORT;
+
+ /** */
+ private int discoPort = TcpDiscoverySpi.DFLT_PORT;
+
+ /** */
+ private byte clusterId = SRC_CLUSTER_ID;
+
+ /** */
+ protected int clientsCnt = 1;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName)
+ .setDiscoverySpi(new TcpDiscoverySpi()
+ .setLocalPort(discoPort)
+ .setIpFinder(new TcpDiscoveryVmIpFinder() {{
+ setAddresses(Collections.singleton("127.0.0.1:" + discoPort + ".." + (discoPort + DFLT_PORT_RANGE)));
+ }}))
+ .setCommunicationSpi(new TcpCommunicationSpi()
+ .setLocalPort(commPort));
Review comment:
Commented this, but tests still green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655494417
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
+ closed.set(true);
Review comment:
`stopped` seems to be more suitable.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/CdcEventsApplier.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.BooleanSupplier;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+
+/**
+ * Contains logic to process {@link ChangeDataCaptureEvent} and apply them to the provided by {@link #ignite()} cluster.
+ */
+public abstract class CdcEventsApplier {
+ /** Maximum batch size. */
+ private final int maxBatchSize;
+
+ /** Caches. */
+ private final Map<Integer, IgniteInternalCache<BinaryObject, BinaryObject>> ignCaches = new HashMap<>();
+
+ /** Update batch. */
+ private final Map<KeyCacheObject, GridCacheDrInfo> updBatch = new HashMap<>();
+
+ /** Remove batch. */
+ private final Map<KeyCacheObject, GridCacheVersion> rmvBatch = new HashMap<>();
+
+ /** */
+ private final BooleanSupplier hasUpdates = () -> !F.isEmpty(updBatch);
+
+ /** */
+ private final BooleanSupplier hasRemoves = () -> !F.isEmpty(rmvBatch);
+
+ /**
+ * @param maxBatchSize Maximum batch size.
+ */
+ public CdcEventsApplier(int maxBatchSize) {
+ this.maxBatchSize = maxBatchSize;
+ }
+
+ /**
+ * @param evts Events to process.
+ * @throws IgniteCheckedException If failed.
+ */
+ protected void apply(Iterable<ChangeDataCaptureEvent> evts) throws IgniteCheckedException {
+ IgniteInternalCache<BinaryObject, BinaryObject> currCache = null;
+
+ for (ChangeDataCaptureEvent evt : evts) {
+ IgniteInternalCache<BinaryObject, BinaryObject> cache = ignCaches.computeIfAbsent(evt.cacheId(), cacheId -> {
+ for (String cacheName : ignite().cacheNames()) {
+ if (CU.cacheId(cacheName) == cacheId) {
+ // IgniteEx#cachex(String) will return null if cache not initialized with regular Ignite#cache(String) call.
+ ignite().cache(cacheName);
+
+ return ignite().cachex(cacheName);
+ }
+ }
+
+ throw new IllegalStateException("Cache with id not found [cacheId=" + cacheId + ']');
+ });
+
+ if (cache != currCache) {
+ applyIf(currCache, hasUpdates, hasRemoves);
+
+ currCache = cache;
+ }
+
+ CacheEntryVersion order = evt.version();
+
+ KeyCacheObject key = new KeyCacheObjectImpl(evt.key(), null, evt.partition());
+
+ if (evt.value() != null) {
+ applyIf(currCache, () -> isApplyBatch(updBatch, key), hasRemoves);
+
+ CacheObject val = new CacheObjectImpl(evt.value(), null);
+
+ updBatch.put(key, new GridCacheDrInfo(val,
+ new GridCacheVersion(order.topologyVersion(), order.order(), order.nodeOrder(), order.clusterId())));
+ }
+ else {
+ applyIf(currCache, hasUpdates, () -> isApplyBatch(rmvBatch, key));
+
+ rmvBatch.put(key,
+ new GridCacheVersion(order.topologyVersion(), order.order(), order.nodeOrder(), order.clusterId()));
+ }
+ }
+
+ if (currCache != null)
+ applyIf(currCache, hasUpdates, hasRemoves);
+ }
+
+ /**
+ * Applies data from {@code updMap} or {@code rmvBatch} to Ignite if required.
Review comment:
What is `updMap`?
Frefactoring unfriendly comment found )
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
Review comment:
seems we're losing the reason here?
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.CdcEventsApplier;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier extends CdcEventsApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
+
+ /** Caches ids to read. */
+ private final Set<Integer> caches;
+
+ /** Consumers. */
+ private final List<KafkaConsumer<Integer, byte[]>> cnsmrs = new ArrayList<>();
+
+ /** */
+ private final AtomicLong rcvdEvts = new AtomicLong();
+
+ /**
+ * @param ign Ignite instance.
+ * @param log Logger.
+ * @param kafkaProps Kafka properties.
+ * @param topic Topic name.
+ * @param kafkaPartFrom Read from partition.
+ * @param kafkaPartTo Read to partition.
+ * @param caches Cache ids.
+ * @param maxBatchSize Maximum batch size.
+ * @param closed Closed flag.
+ */
+ public KafkaToIgniteCdcStreamerApplier(
+ IgniteEx ign,
+ IgniteLogger log,
+ Properties kafkaProps,
+ String topic,
+ int kafkaPartFrom,
+ int kafkaPartTo,
+ Set<Integer> caches,
+ int maxBatchSize,
+ AtomicBoolean closed
+ ) {
+ super(maxBatchSize);
+
+ this.ign = ign;
+ this.kafkaProps = kafkaProps;
+ this.topic = topic;
+ this.kafkaPartFrom = kafkaPartFrom;
+ this.kafkaPartTo = kafkaPartTo;
+ this.caches = caches;
+ this.closed = closed;
+ this.log = log.getLogger(KafkaToIgniteCdcStreamerApplier.class);
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ U.setCurrentIgniteName(ign.name());
+
+ try {
+ for (int kafkaPart = kafkaPartFrom; kafkaPart < kafkaPartTo; kafkaPart++) {
+ KafkaConsumer<Integer, byte[]> cnsmr = new KafkaConsumer<>(kafkaProps);
+
+ cnsmr.assign(Collections.singleton(new TopicPartition(topic, kafkaPart)));
+
+ cnsmrs.add(cnsmr);
+ }
+
+ Iterator<KafkaConsumer<Integer, byte[]>> cnsmrIter = Collections.emptyIterator();
+
+ while (!closed.get()) {
+ if (!cnsmrIter.hasNext())
+ cnsmrIter = cnsmrs.iterator();
+
+ poll(cnsmrIter.next());
+ }
+ }
+ catch (WakeupException e) {
+ if (!closed.get())
+ log.error("Applier wakeup error!", e);
+ }
+ catch (Throwable e) {
+ log.error("Applier error!", e);
+
+ closed.set(true);
Review comment:
all other appliers should be waked up as well.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
+ closed.set(true);
+
+ appliers.forEach(U::closeQuiet);
Review comment:
any reason to call the method which sets `closed.set(true);` line after it set? %)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652537857
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable oldResolveField;
+ Comparable newResolveField;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ oldResolveField = ((BinaryObject)oldVal).field(conflictResolveField);
+ newResolveField = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ oldResolveField = U.field(oldVal, conflictResolveField);
+ newResolveField = U.field(newVal, conflictResolveField);
+ }
+
+ if (oldResolveField == null || newResolveField == null)
+ return oldResolveField == null;
+
+ return oldResolveField.compareTo(newResolveField) < 0;
+ }
+ catch (Exception e) {
Review comment:
as you may know, cache resolver work inside Ignite node.
So yes, node one by one will fail in case incorrect `Comparable` implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r656151300
##########
File path: modules/cdc-ext/src/test/java/org/apache/ignite/cdc/AbstractReplicationTest.java
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.IntStream;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverPluginProvider;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.DFLT_PORT_RANGE;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+@RunWith(Parameterized.class)
+public abstract class AbstractReplicationTest extends GridCommonAbstractTest {
+ /** Cache mode. */
+ @Parameterized.Parameter
+ public CacheAtomicityMode cacheMode;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public int backupCnt;
+
+ /** @return Test parameters. */
+ @Parameterized.Parameters(name = "cacheMode={0},backupCnt={1}")
+ public static Collection<?> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ for (CacheAtomicityMode mode : EnumSet.of(ATOMIC, TRANSACTIONAL))
+ for (int i = 0; i < 2; i++)
+ params.add(new Object[] {mode, i});
+
+ return params;
+ }
+
+ /** */
+ public static final String AP_CACHE = "active-passive-cache";
+
+ /** */
+ public static final String ACTIVE_ACTIVE_CACHE = "active-active-cache";
+
+ /** */
+ public static final byte SRC_CLUSTER_ID = 26;
+
+ /** */
+ public static final byte DEST_CLUSTER_ID = 27;
+
+ /** */
+ public static final int EXISTS = 1;
+
+ /** */
+ public static final int REMOVED = 2;
Review comment:
What's wrong with int constants?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652463756
##########
File path: modules/cdc-ext/modules/core/src/test/config/tests.properties
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
Review comment:
Why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648040508
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureIgniteToKafka.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.resources.LoggerResource;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+/**
+ * Change Data Consumer that streams all data changes to Kafka topic.
+ * {@link ChangeDataCaptureEvent} spread across Kafka topic partitions with {@code {ignite_partition} % {kafka_topic_count}} formula.
+ * Consumer will just fail in case of any error during write. Fail of consumer will lead to the fail of {@code ignite-cdc} application.
+ * It expected that {@code ignite-cdc} will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka unavailability or network issues.
+ *
+ * If you have plans to apply written messages to the other Ignite cluster in active-active manner,
+ * e.g. concurrent updates of the same entry in other cluster is possible,
+ * please, be aware of {@link CacheVersionConflictResolverImpl} conflict resolved.
+ * Configuration of {@link CacheVersionConflictResolverImpl} can be found in {@link ChangeDataCaptureKafkaToIgnite} documentation.
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureKafkaToIgnite
+ * @see CacheVersionConflictResolverImpl
+ */
+public class ChangeDataCaptureIgniteToKafka implements ChangeDataCaptureConsumer {
+ /** Log. */
+ @LoggerResource
+ private IgniteLogger log;
+
+ /** Kafka producer to stream events. */
+ private KafkaProducer<Integer, ChangeDataCaptureEvent> producer;
+
+ /** Handle only primary entry flag. */
+ private final boolean onlyPrimary;
+
+ /** Topic name. */
+ private final String topic;
+
+ /** Number Kafka topic partitions. */
+ private int kafkaPartitionsNum;
Review comment:
kafkaPartsCnt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655230551
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerApplier.java
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.cache.CacheEntryVersion;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.WakeupException;
+
+/**
+ * Thread that polls message from the Kafka topic partitions and applies those messages to the Ignite caches.
+ * It expected that messages was written to the Kafka by the {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * <p>
+ * Each {@code Applier} receive set of Kafka topic partitions to read and caches to process.
+ * Applier creates consumer per partition because Kafka consumer reads not fair,
+ * consumer reads messages from specific partition while there is new messages in specific partition.
+ * See <a href=
+ * "https://cwiki.apache.org/confluence/display/KAFKA/KIP-387%3A+Fair+Message+Consumption+Across+Partitions+in+KafkaConsumer">KIP-387</a>
+ * and <a href="https://issues.apache.org/jira/browse/KAFKA-3932">KAFKA-3932</a> for further information.
+ * All consumers should belongs to the same consumer-group to ensure consistent reading.
+ * Applier polls messages from each consumer in round-robin fashion.
+ * <p>
+ * Messages applied to Ignite using {@link IgniteInternalCache#putAllConflict(Map)}, {@link IgniteInternalCache#removeAllConflict(Map)}
+ * these methods allows to provide {@link GridCacheVersion} of the entry to the Ignite so in case update conflicts they can be resolved
+ * by the {@link CacheVersionConflictResolver}.
+ * <p>
+ * In case of any error during read applier just fail.
+ * Fail of any applier will lead to the fail of {@link KafkaToIgniteCdcStreamer} application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see IgniteToKafkaCdcStreamer
+ * @see IgniteInternalCache#putAllConflict(Map)
+ * @see IgniteInternalCache#removeAllConflict(Map)
+ * @see CacheVersionConflictResolver
+ * @see GridCacheVersion
+ * @see ChangeDataCaptureEvent
+ * @see CacheEntryVersion
+ */
+class KafkaToIgniteCdcStreamerApplier implements Runnable, AutoCloseable {
+ /** */
+ public static final int DFLT_REQ_TIMEOUT = 3;
+
+ /** Ignite instance. */
+ private final IgniteEx ign;
+
+ /** Log. */
+ private final IgniteLogger log;
+
+ /** Closed flag. Shared between all appliers. */
+ private final AtomicBoolean closed;
+
+ /** Caches. */
+ private final Map<Integer, IgniteInternalCache<BinaryObject, BinaryObject>> ignCaches = new HashMap<>();
+
+ /** Kafka properties. */
+ private final Properties kafkaProps;
+
+ /** Maximum batch size. */
+ private final int maxBatchSize;
+
+ /** Topic to read. */
+ private final String topic;
+
+ /** Lower kafka partition (inclusive). */
+ private final int kafkaPartFrom;
+
+ /** Higher kafka partition (exclusive). */
+ private final int kafkaPartTo;
Review comment:
It's not obvious from naming that one is inclusive but another is exclusive.
Lets, at least make both similar.
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Executor service to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final ExecutorService execSvc;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+
+ execSvc = Executors.newFixedThreadPool(streamerCfg.getThreadCount(), new ThreadFactory() {
+ private final AtomicInteger cntr = new AtomicInteger();
+
+ @Override public Thread newThread(@NotNull Runnable r) {
+ Thread th = new Thread(r);
+
+ th.setName("applier-thread-" + cntr.getAndIncrement());
+
+ return th;
+ }
+ });
Review comment:
ExecutorSerice for a single run looks overcomplicated to me.
Why not just Thread.start/join?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r648049096
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureKafkaToIgnite.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.cdc.conflictplugin.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link ChangeDataCaptureIgniteToKafka} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * Properties list:
+ * <ul>
+ * <li>{@link #KAFKA_TO_IGNITE_THREAD_COUNT} - count of {@link Applier} threads.</li>
+ * <li>{@link #IGNITE_TO_KAFKA_TOPIC} - Kafka topic name if not provided in constructor.</li>
+ * </ul>
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class ChangeDataCaptureKafkaToIgnite implements Runnable {
Review comment:
I like the current naming:
`IgniteToKafka` streams changes from Ignite to Kafka.
`KafkaToIgnite` streams changes from Kafka to Ignite.
What's wrong with it?
Please, not, this `Runnable` is a main class of the application.
##########
File path: modules/cdc-to-kafka-ext/src/main/java/org/apache/ignite/cdc/ChangeDataCaptureKafkaToIgnite.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.cdc.conflictplugin.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictplugin.CacheVersionConflictResolverImpl;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.jetbrains.annotations.NotNull;
+
+import static org.apache.ignite.cdc.Utils.property;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link ChangeDataCaptureIgniteToKafka} Change Data Capture consumer.
+ * Application runs several {@link Applier} thread to read Kafka topic partitions and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts(in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * Properties list:
+ * <ul>
+ * <li>{@link #KAFKA_TO_IGNITE_THREAD_COUNT} - count of {@link Applier} threads.</li>
+ * <li>{@link #IGNITE_TO_KAFKA_TOPIC} - Kafka topic name if not provided in constructor.</li>
+ * </ul>
+ *
+ * @see ChangeDataCapture
+ * @see ChangeDataCaptureIgniteToKafka
+ * @see ChangeDataCaptureEvent
+ * @see Applier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class ChangeDataCaptureKafkaToIgnite implements Runnable {
Review comment:
I like the current naming:
`IgniteToKafka` streams changes from Ignite to Kafka.
`KafkaToIgnite` streams changes from Kafka to Ignite.
What's wrong with it?
Please, note, this `Runnable` is a main class of the application.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov merged pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov merged pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652485589
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/conflictresolve/CacheVersionConflictResolverImpl.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.conflictresolve;
+
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * This class implements simple conflict resolution algorithm.
+ * Algorithm decides which version of the entry should be used "new" or "old".
+ * The following steps performed:
+ * <ul>
+ * <li>If entry is freshly created then new version used - {@link GridCacheVersionedEntryEx#isStartVersion()}.</li>
+ * <li>If change made in this cluster then new version used - {@link GridCacheVersionedEntryEx#dataCenterId()}.</li>
+ * <li>If cluster of new entry equal to cluster of old entry
+ * then entry with the greater {@link GridCacheVersionedEntryEx#order()} used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of new entry greater then new version used.</li>
+ * <li>If {@link #conflictResolveField} provided and field of old entry greater then old version used.</li>
+ * <li>Entry with the lower value of {@link GridCacheVersionedEntryEx#dataCenterId()} used.</li>
+ * </ul>
+ *
+ * Note, data center with lower value has greater priority e.g first (1) data center is main in case conflict can't be resolved
+ * automatically.
+ */
+public class CacheVersionConflictResolverImpl implements CacheVersionConflictResolver {
+ /**
+ * Cluster id.
+ * Note, cluster with lower value has greater priority e.g first (1) cluster is main in case conflict can't be resolved automatically.
+ */
+ private final byte clusterId;
+
+ /**
+ * Field for conflict resolve.
+ * Value of this field will be used to compare two entries in case of conflicting changes.
+ * values of this field must implement {@link Comparable} interface.
+ * <pre><i>Note, value of this field used to resolve conflict for external updates only.</i>
+ *
+ * @see CacheVersionConflictResolverImpl
+ */
+ private final String conflictResolveField;
+
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /** If {@code true} then conflict resolving with the value field enabled. */
+ private boolean conflictResolveFieldEnabled;
+
+ /**
+ * @param clusterId Data center id.
+ * @param conflictResolveField Field to resolve conflicts.
+ * @param log Logger.
+ */
+ public CacheVersionConflictResolverImpl(byte clusterId, String conflictResolveField, IgniteLogger log) {
+ this.clusterId = clusterId;
+ this.conflictResolveField = conflictResolveField;
+ this.log = log;
+
+ conflictResolveFieldEnabled = conflictResolveField != null;
+ }
+
+ /** {@inheritDoc} */
+ @Override public <K, V> GridCacheVersionConflictContext<K, V> resolve(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry,
+ boolean atomicVerComparator
+ ) {
+ GridCacheVersionConflictContext<K, V> res = new GridCacheVersionConflictContext<>(ctx, oldEntry, newEntry);
+
+ if (isUseNew(ctx, oldEntry, newEntry))
+ res.useNew();
+ else {
+ log.warning("Skip update due to the conflict [key=" + newEntry.key() + ", fromCluster=" + newEntry.dataCenterId()
+ + ", toCluster=" + oldEntry.dataCenterId() + ']');
+
+ res.useOld();
+ }
+
+ return res;
+ }
+
+ /**
+ * @param ctx Context.
+ * @param oldEntry Old entry.
+ * @param newEntry New entry.
+ * @param <K> Key type.
+ * @param <V> Key type.
+ * @return {@code True} is should use new entry.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private <K, V> boolean isUseNew(
+ CacheObjectValueContext ctx,
+ GridCacheVersionedEntryEx<K, V> oldEntry,
+ GridCacheVersionedEntryEx<K, V> newEntry
+ ) {
+ if (newEntry.dataCenterId() == clusterId) // Update made on the local cluster always win.
+ return true;
+
+ if (oldEntry.isStartVersion()) // New entry.
+ return true;
+
+ if (oldEntry.dataCenterId() == newEntry.dataCenterId())
+ return newEntry.version().compareTo(oldEntry.version()) > 0; // New version from the same cluster.
+
+ if (conflictResolveFieldEnabled) {
+ Object oldVal = oldEntry.value(ctx);
+ Object newVal = newEntry.value(ctx);
+
+ if (oldVal != null && newVal != null) {
+ Comparable o;
+ Comparable n;
+
+ try {
+ if (oldVal instanceof BinaryObject) {
+ o = ((BinaryObject)oldVal).field(conflictResolveField);
+ n = ((BinaryObject)newVal).field(conflictResolveField);
+ }
+ else {
+ o = U.field(oldVal, conflictResolveField);
+ n = U.field(newVal, conflictResolveField);
+ }
+
+ if (o == null || n == null)
+ return o == null;
+
+ return o.compareTo(n) < 0;
+ }
+ catch (Exception e) {
+ log.error("Error while resolving replication conflict with field '" +
+ conflictResolveField + "'.\nConflict resolve with the field disabled.", e);
+
+ conflictResolveFieldEnabled = false;
+ }
+ }
+ }
+
+ // Cluster with the lower ID have biggest priority(e.g. first cluster is main).
+ return newEntry.dataCenterId() < oldEntry.dataCenterId();
Review comment:
> Value of this flag analyzed during toString invocation.
Great!
So, only warn required here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r655530291
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamer.java
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.cdc.ChangeDataCaptureEvent;
+import org.apache.ignite.cdc.conflictresolve.CacheConflictResolutionManagerImpl;
+import org.apache.ignite.cdc.conflictresolve.CacheVersionConflictResolverImpl;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.ChangeDataCapture;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+
+import static org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Main class of Kafka to Ignite application.
+ * This application is counterpart of {@link IgniteToKafkaCdcStreamer} Change Data Capture consumer.
+ * Application runs several {@link KafkaToIgniteCdcStreamerApplier} thread to read Kafka topic partitions
+ * and apply {@link ChangeDataCaptureEvent} to Ignite.
+ * <p>
+ * Each applier receive even number of kafka topic partition to read.
+ * <p>
+ * In case of any error during read applier just fail. Fail of any applier will lead to the fail of whole application.
+ * It expected that application will be configured for automatic restarts with the OS tool to failover temporary errors
+ * such as Kafka or Ignite unavailability.
+ * <p>
+ * To resolve possible update conflicts (in case of concurrent update in source and destination Ignite clusters)
+ * real-world deployments should use some conflict resolver, for example {@link CacheVersionConflictResolverImpl}.
+ * Example of Ignite configuration with the conflict resolver:
+ * <pre>
+ * {@code
+ * CacheVersionConflictResolverCachePluginProvider conflictPlugin = new CacheVersionConflictResolverCachePluginProvider();
+ *
+ * conflictPlugin.setClusterId(clusterId); // Cluster id.
+ * conflictPlugin.setCaches(new HashSet<>(Arrays.asList("my-cache", "some-other-cache"))); // Caches to replicate.
+ *
+ * IgniteConfiguration cfg = ...;
+ *
+ * cfg.setPluginProviders(conflictPlugin);
+ * }
+ * </pre>
+ * Please, see {@link CacheConflictResolutionManagerImpl} for additional information.
+ *
+ * @see ChangeDataCapture
+ * @see IgniteToKafkaCdcStreamer
+ * @see ChangeDataCaptureEvent
+ * @see KafkaToIgniteCdcStreamerApplier
+ * @see CacheConflictResolutionManagerImpl
+ */
+public class KafkaToIgniteCdcStreamer implements Runnable {
+ /** Ignite configuration. */
+ private final IgniteConfiguration igniteCfg;
+
+ /** Kafka consumer properties. */
+ private final Properties kafkaProps;
+
+ /** Streamer configuration. */
+ private final KafkaToIgniteCdcStreamerConfiguration streamerCfg;
+
+ /** Runners to run {@link KafkaToIgniteCdcStreamerApplier} instances. */
+ private final Thread[] runners;
+
+ /** Appliers. */
+ private final List<KafkaToIgniteCdcStreamerApplier> appliers;
+
+ /**
+ * @param igniteCfg Ignite configuration.
+ * @param kafkaProps Kafka properties.
+ * @param streamerCfg Streamer configuration.
+ */
+ public KafkaToIgniteCdcStreamer(
+ IgniteConfiguration igniteCfg,
+ Properties kafkaProps,
+ KafkaToIgniteCdcStreamerConfiguration streamerCfg
+ ) {
+ this.igniteCfg = igniteCfg;
+ this.kafkaProps = kafkaProps;
+ this.streamerCfg = streamerCfg;
+
+ appliers = new ArrayList<>(streamerCfg.getThreadCount());
+ runners = new Thread[streamerCfg.getThreadCount()];
+
+ if (!kafkaProps.containsKey(ConsumerConfig.GROUP_ID_CONFIG))
+ throw new IllegalArgumentException("Kafka properties don't contains " + ConsumerConfig.GROUP_ID_CONFIG);
+
+ kafkaProps.put(KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
+ kafkaProps.put(VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override public void run() {
+ try (IgniteEx ign = (IgniteEx)Ignition.start(igniteCfg)) {
+ IgniteLogger log = U.initLogger(igniteCfg, "kafka-ignite-streamer");
+
+ AtomicBoolean closed = new AtomicBoolean();
+
+ Set<Integer> caches = null;
+
+ if (!F.isEmpty(streamerCfg.getCaches())) {
+ caches = streamerCfg.getCaches().stream()
+ .peek(cache -> Objects.requireNonNull(ign.cache(cache), cache + " not exists!"))
+ .map(CU::cacheId).collect(Collectors.toSet());
+ }
+
+ int kafkaParts = streamerCfg.getKafkaPartitions();
+ int threadCnt = streamerCfg.getThreadCount();
+ int partPerApplier = kafkaParts / threadCnt;
+
+ for (int i = 0; i < threadCnt; i++) {
+ int from = i * partPerApplier;
+ int to = (i + 1) * partPerApplier;
+
+ if (i == threadCnt - 1)
+ to = kafkaParts + 1;
+
+ KafkaToIgniteCdcStreamerApplier applier = new KafkaToIgniteCdcStreamerApplier(
+ ign,
+ log,
+ kafkaProps,
+ streamerCfg.getTopic(),
+ from,
+ to,
+ caches,
+ streamerCfg.getMaxBatchSize(),
+ closed
+ );
+
+ appliers.add(applier);
+
+ runners[i] = new Thread(applier, "applier-thread-" + i);
+
+ runners[i].start();
+ }
+
+ try {
+ for (int i = 0; i < threadCnt; i ++)
+ runners[i].join();
+ }
+ catch (InterruptedException e) {
+ closed.set(true);
+
+ appliers.forEach(U::closeQuiet);
Review comment:
`close` invocation required to call `KafkaConsumer#wakeUp` to cancel current consumer poll request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] anton-vinogradov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r650724395
##########
File path: modules/cdc-ext/src/main/java/org/apache/ignite/cdc/kafka/KafkaToIgniteCdcStreamerConfiguration.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc.kafka;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+
+/**
+ * Configuration of {@link KafkaToIgniteCdcStreamer} application.
+ *
+ * @see KafkaToIgniteCdcStreamer
+ * @see KafkaToIgniteLoader
+ */
+public class KafkaToIgniteCdcStreamerConfiguration {
+ /** Default {@link #kafkaParts} value. */
+ public static final int DFLT_PARTS = 16;
+
+ /** Default {@link #topic} value. */
+ public static final String DFLT_TOPIC = "ignite";
+
+ /** Default {@link #maxBatchSize} value. */
+ public static final int DFLT_MAX_BATCH_SIZE = 1024;
+
+ /** {@link Applier} thread count. */
+ private int threadCnt = DFLT_PARTS;
+
+ /** Topic name. */
+ private String topic = DFLT_TOPIC;
+
+ /** Kafka partitions count. */
+ private int kafkaParts = DFLT_PARTS;
Review comment:
The current logic is not clear to me.
Having `threadCnt` and `kafkaParts` simultaneously with subsequent calculation `partPerApplier = kafkaParts / threadCnt` may set N partitions to some Appliers, but N+1 to other, that's not what user may expect.
We must simplify this to `kafkaParts` and `partPerApplier`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-extensions] nizhikov commented on a change in pull request #49: IGNITE-14362: CDC Ignite to Kafka extension
Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #49:
URL: https://github.com/apache/ignite-extensions/pull/49#discussion_r652480237
##########
File path: modules/cdc-ext/modules/core/src/test/config/tests.properties
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
Review comment:
Please, take a look at `GridTestProperties` static initializer and `GridTestProperties#configureLog4j`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org