You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "tkhurana (via GitHub)" <gi...@apache.org> on 2026/07/06 17:30:59 UTC

[PR] PHOENIX-7874 Eliminate replicating index mutations; regenerate on standby [phoenix]

tkhurana opened a new pull request, #2566:
URL: https://github.com/apache/phoenix/pull/2566

   ## Summary
   
   Index tables no longer generate their own replication-log records. The active ships only the data-table record plus a per-`(row, ts)` **PRE_IMAGE** (the prior row state observed at lock time), and the standby regenerates every index entry from that record on the replay path.
   
   This targets the `PHOENIX-7562-feature-new` feature branch (builds on the per-batch coalescing / cell-oriented log format, #2540).
   
   ## Why
   
   - **Out-of-order replay safety.** Because each replicated batch carries the exact prior-row snapshot it was built against, the standby derives each `(row, ts)` group's next state from its own shipped pre-image instead of a region scan. Shard files can be replayed in any order or in parallel; each group is an independent reproduction of the active `preBatchMutate`.
   - **Cross-cluster rowkeys.** CDC/global/local index rowkeys can embed the encoded data-table region name (`PARTITION_ID`), which differs between clusters. Replicating index cells verbatim would carry the active's region name; the standby instead rebuilds the rowkey with its own.
   
   ## Main changes
   
   - **Cell-oriented replication format** (codec + record + `MutationCellGrouper`) carrying data cells plus one METAFAMILY pre-image cell per row.
   - **`IndexRegionObserver`**: active-side pre-image capture; a forked standby replay path (`prepareReplicatedIndexMutations`) that groups by `(row, ts)` and builds index updates from the shipped pre-image.
   - **`PreImageLocalTable`**: a `LocalHBaseState` serving the local-index builder's prior-row-state from the shipped pre-image instead of a region scan; new `IndexBuildManager.getIndexUpdates` overload accepts it.
   - **`ReplicationLogProcessor`**: replay a record as one indivisible batch; do not retry `DoNotRetryIOException`.
   
   ## Test coverage
   
   Global, local, uncovered, atomic, conditional-TTL, and CDC (plain and behind an EVENTUAL secondary index) index regeneration; concurrent/out-of-order replay for both global and local indexes; unit tests for `(row, ts)` grouping isolation and Put/Delete decomposition.
   
   ## Notes
   
   - Squashed into a single commit.
   - Please squash-merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7874 Eliminate replicating index mutations; regenerate on standby [phoenix]

Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana merged PR #2566:
URL: https://github.com/apache/phoenix/pull/2566


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7874 Eliminate replicating index mutations; regenerate on standby [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2566:
URL: https://github.com/apache/phoenix/pull/2566#discussion_r3618659525


##########
phoenix-core/src/it/java/org/apache/phoenix/replication/ReplicationLogGroupEventualIndexIT.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.replication;
+
+import static org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_INDEX_CDC_CONSUMER_ENABLED;
+import static org.apache.phoenix.query.BaseTest.generateUniqueName;
+
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.util.Arrays;
+import org.apache.phoenix.end2end.NeedsOwnMiniClusterTest;
+import org.apache.phoenix.jdbc.FailoverPhoenixConnection;
+import org.apache.phoenix.util.CDCUtil;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+/**
+ * CDC index behind a CONSISTENCY=EVENTUAL secondary index. Lives in its own IT (not in
+ * {@link ReplicationLogGroupIT}) because the {@code serializeCDCMutations} variant is a
+ * cluster-level config and is exercised by the
+ * {@link ReplicationLogGroupEventualIndexWithSerializeCDCIT} subclass, which inherits exactly this
+ * one test. The base class runs it under the default {@code serializeCDCMutations=false}.
+ */
+@Category(NeedsOwnMiniClusterTest.class)
+public class ReplicationLogGroupEventualIndexIT extends ReplicationLogGroupBaseIT {
+
+  @BeforeClass
+  public static void doSetup() throws Exception {
+    setupEventualIndexClusters();
+  }
+
+  /**
+   * Disables the IndexCDCConsumer on both clusters, then starts them. CDC-index regeneration is
+   * verified directly and the consumer's downstream secondary-index convergence is out of scope, so
+   * it stays off. The {@code serializeCDCMutations} subclass calls this after setting its own
+   * toggle so the consumer-disable lives in one place.
+   */
+  protected static void setupEventualIndexClusters() throws Exception {
+    conf1.setBoolean(PHOENIX_INDEX_CDC_CONSUMER_ENABLED, false);
+    conf2.setBoolean(PHOENIX_INDEX_CDC_CONSUMER_ENABLED, false);

Review Comment:
   do we have coverage for checking if eventually consistent indexes also have rows matching after waiting for some time?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7874 Eliminate replicating index mutations; regenerate on standby [phoenix]

Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana commented on code in PR #2566:
URL: https://github.com/apache/phoenix/pull/2566#discussion_r3624678078


##########
phoenix-core/src/it/java/org/apache/phoenix/replication/ReplicationLogGroupEventualIndexIT.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.replication;
+
+import static org.apache.phoenix.hbase.index.IndexRegionObserver.PHOENIX_INDEX_CDC_CONSUMER_ENABLED;
+import static org.apache.phoenix.query.BaseTest.generateUniqueName;
+
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.util.Arrays;
+import org.apache.phoenix.end2end.NeedsOwnMiniClusterTest;
+import org.apache.phoenix.jdbc.FailoverPhoenixConnection;
+import org.apache.phoenix.util.CDCUtil;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+/**
+ * CDC index behind a CONSISTENCY=EVENTUAL secondary index. Lives in its own IT (not in
+ * {@link ReplicationLogGroupIT}) because the {@code serializeCDCMutations} variant is a
+ * cluster-level config and is exercised by the
+ * {@link ReplicationLogGroupEventualIndexWithSerializeCDCIT} subclass, which inherits exactly this
+ * one test. The base class runs it under the default {@code serializeCDCMutations=false}.
+ */
+@Category(NeedsOwnMiniClusterTest.class)
+public class ReplicationLogGroupEventualIndexIT extends ReplicationLogGroupBaseIT {
+
+  @BeforeClass
+  public static void doSetup() throws Exception {
+    setupEventualIndexClusters();
+  }
+
+  /**
+   * Disables the IndexCDCConsumer on both clusters, then starts them. CDC-index regeneration is
+   * verified directly and the consumer's downstream secondary-index convergence is out of scope, so
+   * it stays off. The {@code serializeCDCMutations} subclass calls this after setting its own
+   * toggle so the consumer-disable lives in one place.
+   */
+  protected static void setupEventualIndexClusters() throws Exception {
+    conf1.setBoolean(PHOENIX_INDEX_CDC_CONSUMER_ENABLED, false);
+    conf2.setBoolean(PHOENIX_INDEX_CDC_CONSUMER_ENABLED, false);

Review Comment:
   @virajjasani Added the test for eventually consistent index



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org