You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "virajjasani (via GitHub)" <gi...@apache.org> on 2026/04/07 21:32:41 UTC
[PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
virajjasani opened a new pull request, #2401:
URL: https://github.com/apache/phoenix/pull/2401
Achieving consistently low latency at any scale is a major challenge for many critical web applications/services. This requires such applications to choose the right database. Distributed NoSQL databases like Apache Phoenix offer the scalability and throughput required for such critical workloads. However, when it comes to global secondary indexes, Phoenix provides strongly consistent (synchronous) indexes. Here, index updates are tightly coupled with the data table updates, meaning as the number of indexes grows, write latency on the data table can increase depending on the network overhead, and/or WAL replica availability of each index table. As a result, applications with high write volumes and multiple indexes can experience some throughput and availability degradation.
The purpose of this Jira is to provide the Eventually Consistent Global Secondary Indexes. Here, index updates are managed separately from the data table updates. This keeps write latency on the data table consistently lower regardless of the number of indexes created on the data table. This allows high write volume applications to take advantage of the global secondary indexes in Phoenix without slowing down their writes, while accepting eventual consistency of the indexes.
The design document attached to the Jira describes several possible approaches to achieve this, while finalizing two approaches to provide eventually consistent indexes.
**Requirements for Eventually Consistent Indexes**
1. Users should be able to create eventually consistent indexes for both covered and uncovered indexes.
2. The SQL statement should include the CONSISTENCY clause to determine whether the given covered or uncovered index is strongly consistent or eventually consistent. By default, consider the given index as strongly consistent. CREATE INDEX <index-name> ON <data-table> ( <col1>,... <colN>) INCLUDE (<col1>,...<colN>) CONSISTENCY = EVENTUAL
3. Users should be able to seamlessly update the CONSISTENCY property of the given index from strong to eventual and vice versa using ALTER INDEX SQL statement. (although the change of consistency update depends on the UPDATE_CACHE_FREQUENCY used at the table level) ALTER INDEX <index-name> ON <data-table> CONSISTENCY = EVENTUAL
4. Depending on the use cases, data tables can consist of the mix of zero or more strongly consistent indexes and zero or more eventually consistent indexes.
5. Index verification MapReduce jobs should work for the eventually consistent global secondary indexes similar to how they work for the strongly consistent global secondary indexes.
6. Concurrent mutations on the data table should work for eventually consistent indexes.
7. Data table mutations need to produce and store the time ordered metadata (change records) for consumers to replay them and perform the index mutation RPCs.
8. Updates to eventually consistent indexes should mirror the pre-index and post-index update semantics of strongly consistent updates. However, the separate RPCs for pre-index and post-index updates can be combined into a single RPC call. For instance, if the data table update failed, the consumer should update corresponding indexes with unverified rows (pre-index updates) only. If the data table update succeeded, the consumer should update corresponding indexes with verified rows (post-index update) only. The consumer does not need to perform both pre and post index update RPCs on the indexes.
9. To improve the scale of index updates, mutations on indexes should be executed by consuming ordered change records per table region. This allows for parallel processing across all table regions.
10. Once the data table region splits or merges into new daughter regions, any remaining ordered change records from the closed parent region should be processed before consuming newly generated change records for the new daughter regions.
[Design doc](https://docs.google.com/document/d/1-bInfjnc0Yg_Aoqs8zc8-HQQzNrKgLtkodEJQxOkOCg/edit?usp=sharing)
Two approaches are finalized for the eventually consistent global secondary indexes:
`phoenix.index.cdc.mutation.serialize` controls which approach is used for implementing eventually consistent global secondary indexes.
**Approach 1: Serialized mutations (value = true)**
During preBatchMutate(), IndexRegionObserver generates index mutations for each data table mutation and serializes them into a Protobuf IndexMutations message. This serialized payload is written as a column value in the CDC index table row alongside the CDC event. The IndexCDCConsumer later reads these pre-computed mutations from the CDC index, deserializes them, and applies them directly to the index table(s). In this approach, the consumer does not need to understand index structure or re-derive mutations — it simply replays what was already computed on the write path. The trade-off is increased CDC index row size due to the serialized mutation payload, and additional write IO on the CDC index table.
**Approach 2: Generated mutations from data row states (default, value = false)**
During preBatchMutate(), IndexRegionObserver writes only a lightweight CDC index entry without serialized index mutations. Instead, the CDC event is created with the DATA_ROW_STATE cdc scope. When the IndexCDCConsumer processes these events, it reads the CDC index rows which trigger a server-side scan of the data table to reconstruct the before-image `currentDataRowState` and after-image `nextDataRowState` of the data row at the change timestamp. These raw row states are returned as a Protobuf DataRowStates message. The consumer then provides these states into generateIndexMutationsForRow() — the same utility used by IndexRegionObserver#prepareIndexMutations on the write path — to derive index mutations at consume time. This approach keeps CDC index rows small, avoids additional write IO, and generates mutations based on the current index definition, but requires an additional data table read per CDC event and is sensitive to data visibility timing. Make sure max lookback age
is long enough to retain before and after images of the row.
Use Approach 2 (serialize = false, default) to minimize write IO: no serialized mutations are written to the CDC index, keeping CDC index rows small and write latency uniform. The trade-off is higher read IO at consume time — the consumer performs an additional data table point-lookup with a raw scan per CDC event to reconstruct row states.
Use Approach 1 (serialize = true) to minimize read IO: the consumer reads pre-computed mutations from the CDC index and applies them directly, with no data table scan required at consume time. The trade-off is higher write IO — serialized index mutations are written alongside each CDC index entry, increasing CDC index row size and write-path latency. Although CDC index is expected to have TTL same as the data table max lookback age.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3083038064
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
Yes it can be, but why we want to make tests more flaky by complicating it with timestamp comparison rather than using simple "wait and compare" approach? I do see more chances of tests getting flaky when we rely on timestamp based comparisons - both in hbase and phoenix. That's why we have tests where we inject custom EnvironmentEdgeManager just to make sure we can expect a specific timestamp. I don't think we need timestamp based comparison. As for phoenix-adapters, the test results comparison is quite simple with wait-and-compare.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082963002
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
> Having this exposed should also help downstream projects like phoenix-adapters to neatly wait for indexes to catch up by polling on the right thing instead of sleeping
I am sure depending on metrics values for testing framework can be as flaky as wait based polling. There is no difference. In fact, it could be more flaky than actually verifying the data correctness in loop.
> We can re-think what more we can expose as a meaningful metric for the consumer in a separate jira.
I did think about it, but I don't see clear useful metric that can help tests. Tests are consistently successful with smaller wait durations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#issuecomment-4274271127
0 test failure in the latest build https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-2401/4/testReport/
<img width="1356" height="624" alt="Screenshot 2026-04-18 at 11 00 38 AM" src="https://github.com/user-attachments/assets/dd5ba73a-7c1e-44b8-879a-71c431ad86b1" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082549295
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
> We can report this metric on the table level as minimum across all regions of this table.
This might need some bookkeeping in `MetricsIndexCDCConsumerSourceImpl` to keep track of `cdcLastEmptyPollTimestamp` across all regions of a table and only publishing the minimum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082921015
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
something like `cdcConsumerLagMs` should be good I guess?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3083021612
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
Do you think some sort of query check on `SYSTEM.IDX_CDC_TRACKER` could also be a helpful signal for tests?
```
SELECT MIN(LAST_TIMESTAMP) FROM SYSTEM.IDX_CDC_TRACKER
WHERE TABLE_NAME = ? AND STATUS = 'i'
```
`MIN(LAST_TIMESTAMP) > afterWritesTimestamp` means consumers have processed everything a test wrote?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3088284363
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
> I wonder how would publishing timestamp help from metrics visualization viewpoint even if we decide to publish it eventually
I think there is some confusion. I agree we do not need to push this as a metric, but using the timestamps from tracker table can be a utility for tests.
> why we want to make tests more flaky by complicating it with timestamp comparison
I think using timestamps from the tracker table should not be flaky in this case. Those timestamps would not be wall clock times, right? Using the tracker table involves querying timestamp values that the consumer itself wrote (the cdc row timestamp), from the same EnvironmentEdgeManager - so there should be no concern of clock skew or different jvm etc. It can be a lightweight pre-check before running the expensive index verification.
But if you think tests are (and will be) stable in CI envs, we can keep this for separate jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082393304
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
Review Comment:
`QueryUtil.getConnectionOnServer` returns JDBC connection, which is lightweight, we use them at many places. Only HConnection is the heavy-weight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082992195
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
I guess the point is we can't get rid of polling but we can decide what to poll on. Verifying data correctness in a loop can be expensive - having a lightweight signal would be really helpful for both phoenix tests and downstream projects. Fine to track as a separate JIRA if not in scope for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082521936
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
Review Comment:
> it will wait for all the current regions of the data table to either get split, merged or moved t
Does that mean the lag in the consumer can be very large in that case?
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
Review Comment:
> it will wait for all the current regions of the data table to either get split, merged or moved
Does that mean the lag in the consumer can be very large in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082836472
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
As far as tests are concerned, I have been running them for very long time https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/tmp-ec/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082925867
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
But as you mentioned in the other thread, we can only get this on the regionserver level right? So it would be something like `cdcConsumerMaxLagMs` which would be the maximum lag of a consumer of a region on a given region server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082965330
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
+ }
+ }
+
+ private String getCdcObjectName(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ throw new SQLException("No CDC object found for table " + dataTableName);
+ }
+ String schemaName = dataTable.getSchemaName().getString();
+ if (schemaName == null || schemaName.isEmpty()) {
+ cdcObjectName = "\"" + cdcObjectName + "\"";
+ } else {
+ cdcObjectName =
+ "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + cdcObjectName + "\"";
+ }
Review Comment:
`SchemaUtil.getQualifiedTableName` could be used here I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082419670
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
+ }
+ }
+
+ private String getCdcObjectName(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ throw new SQLException("No CDC object found for table " + dataTableName);
+ }
+ String schemaName = dataTable.getSchemaName().getString();
+ if (schemaName == null || schemaName.isEmpty()) {
+ cdcObjectName = "\"" + cdcObjectName + "\"";
+ } else {
+ cdcObjectName =
+ "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + cdcObjectName + "\"";
+ }
Review Comment:
Which one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082930339
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
Having this exposed should also help downstream projects like phoenix-adapters to neatly wait for indexes to catch up by polling on the right thing instead of sleeping - WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082823880
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
The value across all regions cannot be provided on the given table without complicating the metrics framework, we don't have such metrics, is that correct? We publish metrics per JVM, not per table.
If this is only for test purpose, we don't need it, we have approach of poll-wait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082941488
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
curious to see this metric in the next perf run!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #2401:
URL: https://github.com/apache/phoenix/pull/2401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082694709
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
Review Comment:
> Does that mean the lag in the consumer can be very large in that case?
Yes, that's the penalty of creating index later. On the other hand, we don't want to refresh PTable asynchronously by making too many syscat rpc calls.
In reality this should not matter because when client creates index later, the index (SC or EC) is always asynchronous. It can take several days to synchronize the index data using mapreduce jobs. So client is not expected to start using index immediately anyway, regardless of the type of the index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082437023
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
Review Comment:
This is interesting one - when EC index is created later, the current region's change logs cannot be processed. Instead, it will wait for all the current regions of the data table to either get split, merged or moved to other server and only after that, the change logs from their parent regions or its own region (from previous server) will start getting processed.
I can add this in the design doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082452042
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentMutationsExtendedIT.java:
##########
@@ -167,6 +182,9 @@ public void run() {
t2.start();
doneSignal.await(60, TimeUnit.SECONDS);
+ if (eventual) {
+ Thread.sleep(35000);
Review Comment:
> Using sleep with large times is going to make tests slow (and flaky but I can guess we cannot do anything about that in this case). Since we are adding multiple new tests which are going to need their own mini cluster, maybe we can have waitFor(predicate, timeout) kind of polling instead to atleast let the test make progress as soon as it can.
I have added such polling in other set of concurrent mutations extended index IT tests, but here I could not add because this test also needs to be used by synchronous index, so we want to keep the changes to the original tests as less as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082974213
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
All the current metrics already do have table level values too. However, they are exposed only from the server:
```
private void incrementTableSpecificHistogram(String baseName, String tableName, long t) {
MetricHistogram tableHistogram =
getMetricsRegistry().getHistogram(getMetricName(baseName, tableName));
tableHistogram.add(t);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082826678
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
Review Comment:
Yes, exactly. In prod, regions split, merge and move all the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082915190
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
Tests maybe be fine today but they need not be in the future or in a different CI env. We already have so many flappers which are mainly due to waiting on async stuff (especially in internal forks). It will be good to avoid adding more such tests. Maybe we can expose this to an in-memory map, which can be consumed in the tests? We can re-think what more we can expose as a meaningful metric for the consumer in a separate jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082406056
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
Yes, let me see if i can incorporate now or as follow-up Jira. Initially, I wanted to avoid adding metrics in this PR but for perf evaluation, checking metrics was also necessary so I added some.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082943166
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
We have metrics for both: per table and per jvm, but per table it cannot be aggregated. Metrics aggregation is only done by various clients.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3083280856
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
done
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
+ }
+ }
+
+ private String getCdcObjectName(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ throw new SQLException("No CDC object found for table " + dataTableName);
+ }
+ String schemaName = dataTable.getSchemaName().getString();
+ if (schemaName == null || schemaName.isEmpty()) {
+ cdcObjectName = "\"" + cdcObjectName + "\"";
+ } else {
+ cdcObjectName =
+ "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + cdcObjectName + "\"";
+ }
+ return cdcObjectName;
+ }
+
+ /**
+ * Processes a batch of CDC events for the given partition starting from the specified timestamp
+ * by generating index mutations from data row states. This method queries the CDC index with the
+ * DATA_ROW_STATE scope, which triggers a server-side data table scan to reconstruct the
+ * before-image ({@code currentDataRowState}) and after-image ({@code nextDataRowState}) for each
+ * change.
+ * @param partitionId the partition (region) ID to process CDC events for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing CDC events from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if a SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatchGenerated(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug(
+ "Processing CDC batch (generated mode) for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(DATA_ROW_STATE) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(DATA_ROW_STATE) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+
+ List<Pair<Long, IndexMutationsProtos.DataRowStates>> batchStates = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ long[] lastScannedTimestamp = { lastProcessedTimestamp };
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchStates.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getDataRowStatesAndTimestamp(ps, newLastTimestamp, batchStates, lastScannedTimestamp);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ if (!batchStates.isEmpty()) {
+ newLastTimestamp = result.getFirst();
+ } else if (retryCount >= maxDataVisibilityRetries) {
+ LOG.warn(
+ "Skipping CDC events for table {} partition {} from timestamp {}"
+ + " to {} after {} retries — data table mutations may have failed",
+ dataTableName, partitionId, newLastTimestamp, lastScannedTimestamp[0], retryCount);
+ newLastTimestamp = lastScannedTimestamp[0];
+ break;
+ } else {
+ // CDC index entries are written but the data is not yet visible.
+ // Don't advance newLastTimestamp so the same events are re-fetched
+ // once the data becomes visible.
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(DATA_ROW_STATE) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchStates.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getDataRowStatesAndTimestamp(ps, newLastTimestamp, batchStates, lastScannedTimestamp);
+ newLastTimestamp = result.getFirst();
+ if (batchStates.isEmpty()) {
+ newLastTimestamp = timestampToRefetch;
+ } else if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ generateAndApplyIndexMutations(conn, batchStates, partitionId, ownerPartitionId,
+ newLastTimestamp);
+ if (!batchStates.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ }
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
done
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
+ }
+ }
+
+ private String getCdcObjectName(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ throw new SQLException("No CDC object found for table " + dataTableName);
+ }
+ String schemaName = dataTable.getSchemaName().getString();
+ if (schemaName == null || schemaName.isEmpty()) {
+ cdcObjectName = "\"" + cdcObjectName + "\"";
+ } else {
+ cdcObjectName =
+ "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + cdcObjectName + "\"";
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082830742
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
Let me publish this, I wonder what would be appropriate name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082965330
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
+ }
+ }
+
+ private String getCdcObjectName(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ throw new SQLException("No CDC object found for table " + dataTableName);
+ }
+ String schemaName = dataTable.getSchemaName().getString();
+ if (schemaName == null || schemaName.isEmpty()) {
+ cdcObjectName = "\"" + cdcObjectName + "\"";
+ } else {
+ cdcObjectName =
+ "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + cdcObjectName + "\"";
+ }
Review Comment:
`SchemaUtil.getQualifiedTableName` could be used here I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082960145
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
Oh okay so we will be able to see the lag for a table as well, that's good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3083290685
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
@palashc we also don't publish timestamp values as metrics usually because it can be bit expensive to represent or visualize? we can publish lag or timestamp diff as histogram, or success/failure events as counts etc. I wonder how would publishing timestamp help even if we decide to publish it eventually, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082507811
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
Having lag would definitely be helpful from the beginning., wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082328846
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
Here we could expose the currentTime in a metric like `cdcLastEmptyPollTimestamp`, the wall-clock time of the last poll that returned no new CDC rows. We can report this metric on the table level as minimum across all regions of this table.
The tests can then use this to decide whether the consumers for a table have caught up. If latest mutation in the test was at time Tm and the minimum `cdcLastEmptyPollTimestamp` for the table is Tc > Tm, then we can say that all consumers have seen the world after the test mutations and found nothing new. This way we can avoid sleeping for large time periods.
I worked out an example of this with Claude, let me know if I am missing something.
Setup
```
10:00:00 UPSERT INTO T VALUES ('Bob', 42) → lands in R1
10:00:00 UPSERT INTO T VALUES ('Zoe', 99) → lands in R2
10:00:00 conn.commit()
10:00:00 afterWrites = now() = 10:00:00
```
Consumer Timeline
```
Time R1 consumer R2 consumer min()
─────────── ──────────────────────────── ──────────────────────────── ──────────
09:59:58 empty poll → reports 09:59:58 empty poll → reports 09:59:58 09:59:58
10:00:00 [writes arrive] [writes arrive]
10:00:01 picks up Bob's CDC row sleeping (poll interval) 09:59:58
10:00:02 processes, writes to index picks up Zoe's CDC row 09:59:58
10:00:03 empty poll → reports 10:00:03 processes, writes to index 09:59:58
10:00:04 sleeping empty poll → reports 10:00:04 10:00:03
```
Test Polling
```
10:00:01 min = 09:59:58 < afterWrites(10:00:00) → keep waiting
10:00:02 min = 09:59:58 < afterWrites(10:00:00) → keep waiting
10:00:03 min = 09:59:58 < afterWrites(10:00:00) → keep waiting (R1 caught up but R2 hasn't)
10:00:04 min = 10:00:03 > afterWrites(10:00:00) → DONE, both caught up
```
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
We should include a metric for lag. We can publish `lag = EnvironmentEdgeManager.currentTimeMillis() - newLastTimestamp` which will be sort if like ageOfLastApplied metric we have in hbase replication. Ofcourse this would include `timestampBufferMs + pollIntervalMs` but if lag goes beyond 11s (using defaults) we will able to tell which consumer might need tuning.
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
+ }
+ }
+
+ private String getCdcObjectName(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ throw new SQLException("No CDC object found for table " + dataTableName);
+ }
+ String schemaName = dataTable.getSchemaName().getString();
+ if (schemaName == null || schemaName.isEmpty()) {
+ cdcObjectName = "\"" + cdcObjectName + "\"";
+ } else {
+ cdcObjectName =
+ "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + cdcObjectName + "\"";
+ }
Review Comment:
Can we use SchemaUtil utility methods?
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
Review Comment:
Comments about what these configs control would be helpful.
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentMutationsExtendedIT.java:
##########
@@ -167,6 +182,9 @@ public void run() {
t2.start();
doneSignal.await(60, TimeUnit.SECONDS);
+ if (eventual) {
+ Thread.sleep(35000);
Review Comment:
Using sleep with large times is going to make tests slow (and flaky but I can guess we cannot do anything about that in this case). Since we are adding multiple new tests which are going to need their own mini cluster, maybe we can have `waitFor(predicate, timeout)` kind of polling instead to atleast let the test make progress as soon as it can.
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
Review Comment:
Any concerns with the connection churn from the CDC consumer since we create a new server side connection every time `QueryUtil.getConnectionOnServer`
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
Review Comment:
What if an EC index is created later? How do we handle that? Should the consumer periodically check instead of returning?
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
+ }
+ }
+
+ private String getCdcObjectName(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ throw new SQLException("No CDC object found for table " + dataTableName);
+ }
+ String schemaName = dataTable.getSchemaName().getString();
+ if (schemaName == null || schemaName.isEmpty()) {
+ cdcObjectName = "\"" + cdcObjectName + "\"";
+ } else {
+ cdcObjectName =
+ "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + cdcObjectName + "\"";
+ }
+ return cdcObjectName;
+ }
+
+ /**
+ * Processes a batch of CDC events for the given partition starting from the specified timestamp
+ * by generating index mutations from data row states. This method queries the CDC index with the
+ * DATA_ROW_STATE scope, which triggers a server-side data table scan to reconstruct the
+ * before-image ({@code currentDataRowState}) and after-image ({@code nextDataRowState}) for each
+ * change.
+ * @param partitionId the partition (region) ID to process CDC events for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing CDC events from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if a SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatchGenerated(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug(
+ "Processing CDC batch (generated mode) for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(DATA_ROW_STATE) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(DATA_ROW_STATE) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+
+ List<Pair<Long, IndexMutationsProtos.DataRowStates>> batchStates = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ long[] lastScannedTimestamp = { lastProcessedTimestamp };
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchStates.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getDataRowStatesAndTimestamp(ps, newLastTimestamp, batchStates, lastScannedTimestamp);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ if (!batchStates.isEmpty()) {
+ newLastTimestamp = result.getFirst();
+ } else if (retryCount >= maxDataVisibilityRetries) {
+ LOG.warn(
+ "Skipping CDC events for table {} partition {} from timestamp {}"
+ + " to {} after {} retries — data table mutations may have failed",
+ dataTableName, partitionId, newLastTimestamp, lastScannedTimestamp[0], retryCount);
+ newLastTimestamp = lastScannedTimestamp[0];
+ break;
+ } else {
+ // CDC index entries are written but the data is not yet visible.
+ // Don't advance newLastTimestamp so the same events are re-fetched
+ // once the data becomes visible.
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(DATA_ROW_STATE) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchStates.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getDataRowStatesAndTimestamp(ps, newLastTimestamp, batchStates, lastScannedTimestamp);
+ newLastTimestamp = result.getFirst();
+ if (batchStates.isEmpty()) {
+ newLastTimestamp = timestampToRefetch;
+ } else if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ generateAndApplyIndexMutations(conn, batchStates, partitionId, ownerPartitionId,
+ newLastTimestamp);
+ if (!batchStates.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ }
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
Review Comment:
Metric for lag can be returned from here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082416760
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
Review Comment:
I have added some Javadoc on configs. Do you think something specific that needs more comments? I wanted to keep naming convention sufficient to understand algorithm because we already have lots of comments throughout all methods and utilities anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082536907
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
Review Comment:
sorry I missed that, all good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082804902
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
Review Comment:
> It can take several days to synchronize the index data using mapreduce jobs.
That's a good point, makes sense. I guess we can keep it this way - you already mentioned you will take care of updating the design for highlighting this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082809790
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
Review Comment:
In production, the gap between index creation and the next region event should be not very huge I guess?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3083290685
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
Review Comment:
@palashc we also don't publish timestamp values as metrics usually because it can be bit expensive to represent or visualize? we can publish lag or timestamp diff as histogram, or success/failure events as counts etc. I wonder how would publishing timestamp help from metrics visualization viewpoint even if we decide to publish it eventually, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7794 Eventually Consistent Global Secondary Indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2401:
URL: https://github.com/apache/phoenix/pull/2401#discussion_r3082972900
##########
phoenix-core-server/src/main/java/org/apache/phoenix/hbase/index/IndexCDCConsumer.java:
##########
@@ -0,0 +1,1349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hbase.index;
+
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.ConnectionUtils;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment;
+import org.apache.phoenix.coprocessor.generated.IndexMutationsProtos;
+import org.apache.phoenix.coprocessorclient.MetaDataProtocol;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexCDCConsumerSource;
+import org.apache.phoenix.hbase.index.metrics.MetricsIndexerSourceFactory;
+import org.apache.phoenix.hbase.index.table.HTableInterfaceReference;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.hbase.index.write.IndexWriter;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.TableNotFoundException;
+import org.apache.phoenix.schema.types.IndexConsistency;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.ByteUtil;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.ServerUtil.ConnectionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.thirdparty.com.google.common.collect.ArrayListMultimap;
+import org.apache.phoenix.thirdparty.com.google.common.collect.ListMultimap;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
+
+/**
+ * A single-threaded background consumer that processes CDC mutations for eventually consistent
+ * indexes. This consumer reads mutations from the CDC index table and applies them to the
+ * appropriate secondary indexes.
+ * <p>
+ * The consumer tracks its progress in the SYSTEM.IDX_CDC_TRACKER table, allowing for proper
+ * handling of region splits and merges.
+ * </p>
+ * <p>
+ * <b>Lifecycle:</b>
+ * <ol>
+ * <li>On startup, the consumer first replays any remaining mutations from parent regions (if this
+ * region was created from a split or merge). A region will have multiple parent regions in case of
+ * region merge.</li>
+ * <li>After replaying all parent mutations, the consumer marks each parent region as COMPLETE in
+ * SYSTEM.IDX_CDC_TRACKER.</li>
+ * <li>The consumer then begins processing mutations for the current region. With every batch of
+ * rows consumed (e.g., 1000 rows), it updates LAST_TIMESTAMP in SYSTEM.IDX_CDC_TRACKER. The tracker
+ * entry with status IN_PROGRESS is created on the first batch update.</li>
+ * <li>When the region is closed (due to split, merge, or shutdown), the consumer stops but does NOT
+ * mark itself as COMPLETE. The COMPLETE status is only set by child regions after they have
+ * replayed all parent mutations.</li>
+ * </ol>
+ * </p>
+ */
+public class IndexCDCConsumer implements Runnable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IndexCDCConsumer.class);
+
+ public static final String INDEX_CDC_CONSUMER_BATCH_SIZE =
+ "phoenix.index.cdc.consumer.batch.size";
+ private static final int DEFAULT_CDC_BATCH_SIZE = 100;
+
+ public static final String INDEX_CDC_CONSUMER_STARTUP_DELAY_MS =
+ "phoenix.index.cdc.consumer.startup.delay.ms";
+ private static final long DEFAULT_STARTUP_DELAY_MS = 10000;
+
+ /**
+ * The interval in milliseconds between processing batches when mutations are found.
+ */
+ public static final String INDEX_CDC_CONSUMER_POLL_INTERVAL_MS =
+ "phoenix.index.cdc.consumer.poll.interval.ms";
+ private static final long DEFAULT_POLL_INTERVAL_MS = 1000;
+
+ /**
+ * The time buffer in milliseconds subtracted from current time when querying CDC mutations to
+ * help avoid reading mutations that are too recent.
+ */
+ public static final String INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS =
+ "phoenix.index.cdc.consumer.timestamp.buffer.ms";
+ private static final long DEFAULT_TIMESTAMP_BUFFER_MS = 10000;
+
+ /**
+ * Maximum number of retries when CDC events exist but the corresponding data table mutations are
+ * not yet visible (or permanently failed). After exceeding this limit, the consumer advances past
+ * the unprocessable events to avoid blocking indefinitely. This is only used for index mutation
+ * generation approach (serializeCDCMutations = false).
+ */
+ public static final String INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES =
+ "phoenix.index.cdc.consumer.max.data.visibility.retries";
+ private static final int DEFAULT_MAX_DATA_VISIBILITY_RETRIES = 15;
+
+ public static final String INDEX_CDC_CONSUMER_RETRY_PAUSE_MS =
+ "phoenix.index.cdc.consumer.retry.pause.ms";
+ private static final long DEFAULT_RETRY_PAUSE_MS = 2000;
+
+ public static final String INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS =
+ "phoenix.index.cdc.consumer.parent.progress.pause.ms";
+ private static final long DEFAULT_PARENT_PROGRESS_PAUSE_MS = 15000;
+
+ private final RegionCoprocessorEnvironment env;
+ private final String dataTableName;
+ private final String encodedRegionName;
+ private final IndexWriter indexWriter;
+ private final long pause;
+ private final long startupDelayMs;
+ private final int batchSize;
+ private final long pollIntervalMs;
+ private final long timestampBufferMs;
+ private final int maxDataVisibilityRetries;
+ private final long parentProgressPauseMs;
+ private final Configuration config;
+ private final boolean serializeCDCMutations;
+ private final MetricsIndexCDCConsumerSource metricSource;
+ private volatile boolean stopped = false;
+ private Thread consumerThread;
+ private boolean hasParentPartitions = false;
+ private PTable cachedDataTable;
+
+ private boolean tenantInit = false;
+ private boolean isMultiTenant = false;
+ private String tenantIdColName;
+ private PDataType<?> tenantIdDataType;
+ private TenantScanInfo ownRegionScanInfo;
+
+ private final Map<String, TenantScanInfo> ancestorScanInfoCache = new HashMap<>();
+
+ private static class TenantScanInfo {
+
+ private static final TenantScanInfo EMPTY = new TenantScanInfo("", "", null, null, null);
+
+ private final String filter;
+ private final String orderBy;
+ private final Object startValue;
+ private final Object endValue;
+ private final PDataType<?> dataType;
+
+ TenantScanInfo(String filter, String orderBy, Object startValue, Object endValue,
+ PDataType<?> dataType) {
+ this.filter = filter;
+ this.orderBy = orderBy;
+ this.startValue = startValue;
+ this.endValue = endValue;
+ this.dataType = dataType;
+ }
+
+ int bindParams(PreparedStatement ps, int startIndex) throws SQLException {
+ int idx = startIndex;
+ if (startValue != null) {
+ ps.setObject(idx++, startValue, dataType.getSqlType());
+ }
+ if (endValue != null) {
+ ps.setObject(idx++, endValue, dataType.getSqlType());
+ }
+ return idx;
+ }
+ }
+
+ /**
+ * Creates a new IndexCDCConsumer for the given region with configurable serialization mode.
+ * @param env region coprocessor environment.
+ * @param dataTableName name of the data table.
+ * @param serverName server name.
+ * @param serializeCDCMutations when true, consumes pre-serialized index mutations; when false,
+ * generates index mutations from data row states.
+ * @throws IOException if the IndexWriter cannot be created.
+ */
+ public IndexCDCConsumer(RegionCoprocessorEnvironment env, String dataTableName, String serverName,
+ boolean serializeCDCMutations) throws IOException {
+ this.env = env;
+ this.dataTableName = dataTableName;
+ this.encodedRegionName = env.getRegion().getRegionInfo().getEncodedName();
+ this.config = env.getConfiguration();
+ this.serializeCDCMutations = serializeCDCMutations;
+ this.pause = config.getLong(INDEX_CDC_CONSUMER_RETRY_PAUSE_MS, DEFAULT_RETRY_PAUSE_MS);
+ this.startupDelayMs =
+ config.getLong(INDEX_CDC_CONSUMER_STARTUP_DELAY_MS, DEFAULT_STARTUP_DELAY_MS);
+ int baseBatchSize = config.getInt(INDEX_CDC_CONSUMER_BATCH_SIZE, DEFAULT_CDC_BATCH_SIZE);
+ int jitter = ThreadLocalRandom.current().nextInt(baseBatchSize / 5 + 1);
+ this.batchSize = baseBatchSize + jitter;
+ this.pollIntervalMs =
+ config.getLong(INDEX_CDC_CONSUMER_POLL_INTERVAL_MS, DEFAULT_POLL_INTERVAL_MS);
+ this.timestampBufferMs =
+ config.getLong(INDEX_CDC_CONSUMER_TIMESTAMP_BUFFER_MS, DEFAULT_TIMESTAMP_BUFFER_MS);
+ this.maxDataVisibilityRetries = config.getInt(INDEX_CDC_CONSUMER_MAX_DATA_VISIBILITY_RETRIES,
+ DEFAULT_MAX_DATA_VISIBILITY_RETRIES);
+ this.parentProgressPauseMs =
+ config.getLong(INDEX_CDC_CONSUMER_PARENT_PROGRESS_PAUSE_MS, DEFAULT_PARENT_PROGRESS_PAUSE_MS);
+ this.metricSource = MetricsIndexerSourceFactory.getInstance().getIndexCDCConsumerSource();
+ DelegateRegionCoprocessorEnvironment indexWriterEnv =
+ new DelegateRegionCoprocessorEnvironment(env, ConnectionType.INDEX_WRITER_CONNECTION);
+ this.indexWriter =
+ new IndexWriter(indexWriterEnv, serverName + "-index-eventual-writer", false);
+ }
+
+ /**
+ * Starts the consumer thread in the background.
+ */
+ public void start() {
+ consumerThread =
+ new Thread(this, "IndexCDCConsumer-" + dataTableName + "-" + encodedRegionName);
+ consumerThread.setDaemon(true);
+ consumerThread.start();
+ }
+
+ /**
+ * Stops the consumer thread gracefully.
+ */
+ public void stop() {
+ stopped = true;
+ if (consumerThread != null) {
+ consumerThread.interrupt();
+ }
+ if (indexWriter != null) {
+ indexWriter.stop("IndexCDCConsumer stopped for " + dataTableName);
+ }
+ }
+
+ /**
+ * Sleeps for the specified duration if the consumer has not been stopped.
+ * @param millis the duration to sleep in milliseconds.
+ * @throws InterruptedException if the thread is interrupted while sleeping.
+ */
+ private void sleepIfNotStopped(long millis) throws InterruptedException {
+ if (!stopped) {
+ Thread.sleep(millis);
+ }
+ }
+
+ private PTable getDataTable(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = cachedDataTable;
+ if (dataTable == null) {
+ dataTable = conn.getTable(dataTableName);
+ cachedDataTable = dataTable;
+ }
+ return dataTable;
+ }
+
+ private void refreshDataTableCache(PhoenixConnection conn) throws SQLException {
+ cachedDataTable = conn.getTable(dataTableName);
+ }
+
+ private void initTenantInfo(PhoenixConnection conn) throws SQLException {
+ if (tenantInit) {
+ return;
+ }
+ PTable dataTable = getDataTable(conn);
+ isMultiTenant = dataTable.isMultiTenant();
+ if (!isMultiTenant) {
+ ownRegionScanInfo = TenantScanInfo.EMPTY;
+ tenantInit = true;
+ return;
+ }
+ int tenantColIndex = dataTable.getBucketNum() != null ? 1 : 0;
+ PColumn tenantCol = dataTable.getPKColumns().get(tenantColIndex);
+ tenantIdColName = tenantCol.getName().getString();
+ tenantIdDataType = tenantCol.getDataType();
+
+ byte[] regionStartKey = env.getRegion().getRegionInfo().getStartKey();
+ byte[] regionEndKey = env.getRegion().getRegionInfo().getEndKey();
+ ownRegionScanInfo = buildTenantScanInfo(regionStartKey, regionEndKey, dataTable);
+ LOG.debug(
+ "Initialized multi-tenant scan for table {} region {}:"
+ + " tenantCol {}, startTenant {}, endTenant {}",
+ dataTableName, encodedRegionName, tenantIdColName, ownRegionScanInfo.startValue,
+ ownRegionScanInfo.endValue);
+ tenantInit = true;
+ }
+
+ private TenantScanInfo buildTenantScanInfo(byte[] startKey, byte[] endKey, PTable dataTable) {
+ Object startVal = extractTenantIdFromRegionKey(startKey, dataTable);
+ Object endVal = extractTenantIdFromRegionKey(endKey, dataTable);
+ StringBuilder sb = new StringBuilder();
+ if (startVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" >= ? AND ");
+ }
+ if (endVal != null) {
+ sb.append("\"").append(tenantIdColName).append("\" <= ? AND ");
+ }
+ String filter = sb.toString();
+ String orderBy = filter.isEmpty() ? "" : "\"" + tenantIdColName + "\" ASC,";
+ return new TenantScanInfo(filter, orderBy, startVal, endVal, tenantIdDataType);
+ }
+
+ private Object extractTenantIdFromRegionKey(byte[] regionKey, PTable dataTable) {
+ if (regionKey == null || regionKey.length == 0) {
+ return null;
+ }
+ final RowKeySchema schema = dataTable.getRowKeySchema();
+ int pkPos = dataTable.getBucketNum() != null ? 1 : 0;
+ ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+ int maxOffset = schema.iterator(regionKey, 0, regionKey.length, ptr);
+ for (int i = 0; i <= pkPos; i++) {
+ Boolean hasValue = schema.next(ptr, i, maxOffset);
+ if (!Boolean.TRUE.equals(hasValue)) {
+ return null;
+ }
+ }
+ byte[] tenantBytes = ByteUtil.copyKeyBytesIfNecessary(ptr);
+ PColumn tenantCol = dataTable.getPKColumns().get(pkPos);
+ return tenantCol.getDataType().toObject(tenantBytes, 0, tenantBytes.length,
+ tenantCol.getDataType(), tenantCol.getSortOrder(), tenantCol.getMaxLength(),
+ tenantCol.getScale());
+ }
+
+ private byte[][] lookupPartitionKeys(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ final String query = "SELECT PARTITION_START_KEY, PARTITION_END_KEY FROM "
+ + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ byte[] startKey = rs.getBytes(1);
+ byte[] endKey = rs.getBytes(2);
+ return new byte[][] { startKey == null ? new byte[0] : startKey,
+ endKey == null ? new byte[0] : endKey };
+ }
+ }
+ LOG.error("No CDC_STREAM entry found for partition {} table {}. This should not happen.",
+ partitionId, dataTableName);
+ return new byte[][] { new byte[0], new byte[0] };
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error while retrieving partition keys from CDC_STREAM for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return null;
+ }
+
+ private TenantScanInfo getPartitionTenantScanInfo(String partitionId)
+ throws InterruptedException {
+ if (!isMultiTenant) {
+ return TenantScanInfo.EMPTY;
+ }
+ if (partitionId.equals(encodedRegionName)) {
+ return ownRegionScanInfo;
+ }
+ TenantScanInfo cached = ancestorScanInfoCache.get(partitionId);
+ if (cached != null) {
+ return cached;
+ }
+ byte[][] keys = lookupPartitionKeys(partitionId);
+ if (keys == null) {
+ return TenantScanInfo.EMPTY;
+ }
+ TenantScanInfo info = buildTenantScanInfo(keys[0], keys[1], cachedDataTable);
+ ancestorScanInfoCache.put(partitionId, info);
+ return info;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (startupDelayMs > 0 && getCDCStreamNumPartitions() <= 1) {
+ sleepIfNotStopped(startupDelayMs);
+ }
+ if (stopped) {
+ return;
+ }
+ if (!hasEventuallyConsistentIndexes()) {
+ LOG.trace("No eventually consistent indexes found for table {}. Exiting consumer.",
+ dataTableName);
+ return;
+ }
+ LOG.info(
+ "IndexCDCConsumer started for table {} region {}"
+ + " [batchSize: {}, pollIntervalMs: {}, timestampBufferMs: {}, startupDelayMs: {},"
+ + " pause: {}, maxDataVisibilityRetries: {}, parentProgressPauseMs: {},"
+ + " serializeCDCMutations: {}]",
+ dataTableName, encodedRegionName, batchSize, pollIntervalMs, timestampBufferMs,
+ startupDelayMs, pause, maxDataVisibilityRetries, parentProgressPauseMs,
+ serializeCDCMutations);
+ if (!waitForCDCStreamEntry()) {
+ LOG.error(
+ "IndexCDCConsumer stopped while waiting for CDC_STREAM entry for table {} region {}",
+ dataTableName, encodedRegionName);
+ return;
+ }
+ long lastProcessedTimestamp = checkTrackerStatus(encodedRegionName, encodedRegionName);
+ if (lastProcessedTimestamp == -1) {
+ // should never happen as COMPLETE is only set by child regions for their parent regions
+ LOG.error(
+ "Unexpected COMPLETE status in IDX_CDC_TRACKER for current region. "
+ + "Table: {}, Partition: {}. Exiting consumer thread.",
+ dataTableName, encodedRegionName);
+ return;
+ } else if (lastProcessedTimestamp > 0) {
+ LOG.info(
+ "Found existing tracker entry for table {} region {} with lastTimestamp {}. "
+ + "Resuming from last position (region movement scenario).",
+ dataTableName, encodedRegionName, lastProcessedTimestamp);
+ } else {
+ if (hasParentPartitions) {
+ sleepIfNotStopped(timestampBufferMs + 1);
+ replayAndCompleteParentRegions(encodedRegionName);
+ } else {
+ LOG.info("No parent partitions for table {} region {}, skipping parent replay",
+ dataTableName, encodedRegionName);
+ }
+ }
+ int retryCount = 0;
+ while (!stopped) {
+ try {
+ long previousTimestamp = lastProcessedTimestamp;
+ if (serializeCDCMutations) {
+ lastProcessedTimestamp =
+ processCDCBatch(encodedRegionName, encodedRegionName, lastProcessedTimestamp, false);
+ } else {
+ lastProcessedTimestamp = processCDCBatchGenerated(encodedRegionName, encodedRegionName,
+ lastProcessedTimestamp, false);
+ }
+ if (lastProcessedTimestamp == previousTimestamp) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ } else {
+ retryCount = 0;
+ sleepIfNotStopped(pollIntervalMs);
+ }
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ throw (InterruptedException) e;
+ }
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.error(
+ "Error processing CDC mutations for table {} region {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ } catch (InterruptedException e) {
+ if (!stopped) {
+ LOG.warn("IndexCDCConsumer interrupted unexpectedly for table {} region {}", dataTableName,
+ encodedRegionName, e);
+ }
+ Thread.currentThread().interrupt();
+ }
+ LOG.info("IndexCDCConsumer exiting for table {} region {}", dataTableName, encodedRegionName);
+ }
+
+ private boolean hasEventuallyConsistentIndexes() throws InterruptedException {
+ int retryCount = 0;
+ while (!stopped) {
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ refreshDataTableCache(conn);
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ LOG.debug("No CDC index found for table {}. Exiting consumer.", dataTableName);
+ return false;
+ }
+ for (PTable index : dataTable.getIndexes()) {
+ IndexConsistency consistency = index.getIndexConsistency();
+ if (consistency != null && consistency.isAsynchronous()) {
+ LOG.debug("Found eventually consistent index {} for table {}",
+ index.getName().getString(), dataTableName);
+ return true;
+ }
+ }
+ return false;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking for eventually consistent indexes for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Retrieves the count of partitions for the given table.
+ * @return the count of CDC_STREAM rows for this table.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getCDCStreamNumPartitions() throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT COUNT(*) FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ return rs.getLong(1);
+ }
+ }
+ return 0;
+ } catch (SQLException e) {
+ if (e instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) e;
+ // 5.3.0+ server with old metadata tables (i.e. EXECUTE_UPGRADE is not yet run)
+ if (PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_TABLE.equals(tnfe.getTableName())) {
+ stopped = true;
+ return -1;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting CDC_STREAM row count for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Waits for the CDC_STREAM entry for the given table and partition to be available.
+ * @return true if the entry was found, false if the consumer was stopped before finding the
+ * entry.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean waitForCDCStreamEntry() throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? LIMIT 1";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, encodedRegionName);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ String parentPartitionId = rs.getString(1);
+ hasParentPartitions = parentPartitionId != null;
+ LOG.debug("Found CDC_STREAM entry for table {} partition {}, hasParentPartitions={}",
+ dataTableName, encodedRegionName, hasParentPartitions);
+ return true;
+ }
+ }
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.info(
+ "CDC_STREAM entry not found for table {} partition {}. "
+ + "Attempt #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime);
+ sleepIfNotStopped(sleepTime);
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, encodedRegionName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks for an existing entry for the given partition and owner.
+ * @param partitionId the partition ID to check.
+ * @param ownerPartitionId the owner partition ID.
+ * @return the last processed timestamp if an entry exists with status IN_PROGRESS, -1 if the
+ * entry exists with status COMPLETE, or 0 if no entry exists.
+ * @throws InterruptedException if the thread is interrupted while waiting
+ */
+ private long checkTrackerStatus(String partitionId, String ownerPartitionId)
+ throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT LAST_TIMESTAMP, STATUS FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND OWNER_PARTITION_ID = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, ownerPartitionId);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long lastTimestamp = rs.getLong(1);
+ String status = rs.getString(2);
+ LOG.debug(
+ "Found IDX_CDC_TRACKER entry for table {} partition {} owner {} "
+ + "with status={}, lastTimestamp={}",
+ dataTableName, partitionId, ownerPartitionId, status, lastTimestamp);
+ if (PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS.equals(status)) {
+ return lastTimestamp;
+ } else if (PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE.equals(status)) {
+ return -1;
+ } else {
+ // Unknown status - should not happen
+ throw new IllegalStateException(
+ String.format("Unknown tracker status '%s' for table %s partition %s owner %s.",
+ status, dataTableName, partitionId, ownerPartitionId));
+ }
+ } else {
+ LOG.debug("No IDX_CDC_TRACKER entry found for table {} partition {} owner {}",
+ dataTableName, partitionId, ownerPartitionId);
+ return 0;
+ }
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking IDX_CDC_TRACKER for table {} partition {} owner {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, ownerPartitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Checks if any consumer has already completed processing the given partition.
+ * @param partitionId the partition ID to check
+ * @return true if the partition is done processing.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private boolean isPartitionCompleted(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query = "SELECT * FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ try (ResultSet rs = ps.executeQuery()) {
+ return rs.next();
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error checking if partition {} is completed for table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Gets the maximum last processed timestamp from any consumer's tracker record for the given
+ * partition. This is used when a child region needs to continue processing from where any
+ * previous consumer left off.
+ * @param partitionId the partition ID to get progress for.
+ * @return the max last processed timestamp across all owners, or 0 if not found.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long getParentProgress(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT MAX(LAST_TIMESTAMP) FROM " + PhoenixDatabaseMetaData.SYSTEM_IDX_CDC_TRACKER_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND STATUS = ?";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ ps.setString(3, PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ try (ResultSet rs = ps.executeQuery()) {
+ if (rs.next()) {
+ long maxTimestamp = rs.getLong(1);
+ if (maxTimestamp > 0) {
+ LOG.debug("Found max progress {} for partition {} from previous consumers",
+ maxTimestamp, partitionId);
+ return maxTimestamp;
+ }
+ }
+ return 0;
+ }
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error getting parent progress for partition {} table {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ partitionId, dataTableName, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ throw new InterruptedException("IndexCDCConsumer stopped while getting parent progress.");
+ }
+
+ /**
+ * Retrieves all parent partition IDs for the given partition.
+ * @param partitionId the partition ID to find parents for.
+ * @return list of parent partition IDs.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private List<String> getParentPartitionIds(String partitionId) throws InterruptedException {
+ int retryCount = 0;
+ String query =
+ "SELECT PARENT_PARTITION_ID FROM " + PhoenixDatabaseMetaData.SYSTEM_CDC_STREAM_NAME
+ + " WHERE TABLE_NAME = ? AND PARTITION_ID = ? AND PARENT_PARTITION_ID IS NOT NULL";
+ while (!stopped) {
+ try (
+ PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(env.getConfiguration()).unwrap(PhoenixConnection.class);
+ PreparedStatement ps = conn.prepareStatement(query)) {
+ ps.setString(1, dataTableName);
+ ps.setString(2, partitionId);
+ List<String> parentIds = new ArrayList<>();
+ try (ResultSet rs = ps.executeQuery()) {
+ while (rs.next()) {
+ parentIds.add(rs.getString(1));
+ }
+ }
+ LOG.debug("Found {} parent partition(s) for table {} partition {}: {}", parentIds.size(),
+ dataTableName, partitionId, parentIds);
+ return parentIds;
+ } catch (SQLException e) {
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error querying parent partitions from CDC_STREAM for table {} partition {}. "
+ + "Retry #{}, sleeping {} ms before retrying...",
+ dataTableName, partitionId, retryCount, sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ /**
+ * Processes all remaining CDC mutations for the given partition until completion, then marks the
+ * partition as COMPLETE. Used to replay ancestor partitions (regions) after they are split or
+ * merged into new regions.
+ * @param partitionId the partition ID to process.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start processing from.
+ * @throws InterruptedException if the thread is interrupted.
+ */
+ private void processPartitionToCompletion(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp) throws InterruptedException {
+ LOG.debug(
+ "Processing partition {} owner {} to completion for table {}, starting from timestamp {}",
+ partitionId, ownerPartitionId, dataTableName, lastProcessedTimestamp);
+ long currentLastProcessedTimestamp = lastProcessedTimestamp;
+ int retryCount = 0;
+ int batchCount = 0;
+ while (!stopped) {
+ try {
+ if (batchCount > 0) {
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ long otherProgress = getParentProgress(partitionId);
+ if (otherProgress > currentLastProcessedTimestamp) {
+ long previousOtherProgress;
+ do {
+ previousOtherProgress = otherProgress;
+ sleepIfNotStopped(parentProgressPauseMs);
+ if (isPartitionCompleted(partitionId)) {
+ return;
+ }
+ otherProgress = getParentProgress(partitionId);
+ } while (!stopped && otherProgress > previousOtherProgress);
+ currentLastProcessedTimestamp = otherProgress;
+ }
+ }
+ long newTimestamp;
+ if (serializeCDCMutations) {
+ newTimestamp =
+ processCDCBatch(partitionId, ownerPartitionId, currentLastProcessedTimestamp, true);
+ } else {
+ newTimestamp = processCDCBatchGenerated(partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, true);
+ }
+ batchCount++;
+ retryCount = 0;
+ if (newTimestamp == currentLastProcessedTimestamp) {
+ if (isPartitionCompleted(partitionId)) {
+ LOG.info(
+ "Partition {} for table {} was completed by another consumer before {} could mark it",
+ partitionId, dataTableName, ownerPartitionId);
+ return;
+ }
+ LOG.info("Partition {} owner {} for table {} fully processed, marking as COMPLETE",
+ partitionId, ownerPartitionId, dataTableName);
+ try (PhoenixConnection conn = QueryUtil.getConnectionOnServer(env.getConfiguration())
+ .unwrap(PhoenixConnection.class)) {
+ updateTrackerProgress(conn, partitionId, ownerPartitionId,
+ currentLastProcessedTimestamp, PhoenixDatabaseMetaData.TRACKER_STATUS_COMPLETE);
+ }
+ return;
+ }
+ currentLastProcessedTimestamp = newTimestamp;
+ } catch (SQLException | IOException e) {
+ metricSource.incrementCdcBatchFailureCount(dataTableName);
+ long sleepTime = ConnectionUtils.getPauseTime(pause, ++retryCount);
+ LOG.warn(
+ "Error processing CDC batch for partition {} owner {} table {} "
+ + "lastProcessedTimestamp {}. Retry #{}, sleeping {} ms",
+ partitionId, ownerPartitionId, dataTableName, currentLastProcessedTimestamp, retryCount,
+ sleepTime, e);
+ sleepIfNotStopped(sleepTime);
+ }
+ }
+ LOG.info("Processing partition {} (owner {}) stopped before completion for table {}",
+ partitionId, ownerPartitionId, dataTableName);
+ }
+
+ /**
+ * Recursive helper method to replay and complete parent regions. Replays all remaining mutations
+ * from parent regions and marks them as COMPLETE. This is called during initialization when a
+ * region has parent regions (split/merge scenarios).
+ * @param partitionId the partition (region) ID to find and process parents for.
+ */
+ private void replayAndCompleteParentRegions(String partitionId) throws InterruptedException {
+ List<String> parentIds = getParentPartitionIds(partitionId);
+ if (parentIds.isEmpty()) {
+ LOG.debug("No parent partitions found for table {} partition {}", dataTableName, partitionId);
+ return;
+ }
+ for (String parentId : parentIds) {
+ if (stopped) {
+ return;
+ }
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug("Parent partition {} for table {} already has a COMPLETE record, skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ replayAndCompleteParentRegions(parentId);
+ if (isPartitionCompleted(parentId)) {
+ LOG.debug(
+ "Parent partition {} for table {} was completed by sibling during ancestor processing, "
+ + "skipping",
+ parentId, dataTableName);
+ continue;
+ }
+ long parentProgress = getParentProgress(parentId);
+ LOG.debug("Processing/Resuming parent partition {} for table {} from timestamp {} owner: {}",
+ parentId, dataTableName, parentProgress, encodedRegionName);
+ processPartitionToCompletion(parentId, encodedRegionName, parentProgress);
+ }
+ }
+
+ /**
+ * Processes a batch of CDC mutations for the given partition starting from the specified
+ * timestamp. This method reads mutations from the CDC index, applies them to the respective
+ * eventually consistent index tables, and updates the progress in SYSTEM.IDX_CDC_TRACKER.
+ * @param partitionId the partition (region) ID to process CDC mutations for.
+ * @param ownerPartitionId the owner partition ID.
+ * @param lastProcessedTimestamp the timestamp to start reading CDC mutations from.
+ * @param isParentReplay true if replaying a closed parent partition.
+ * @return the new last processed timestamp after this batch, or the same timestamp if no new
+ * records were found.
+ * @throws SQLException if SQL error occurs.
+ * @throws IOException if an I/O error occurs.
+ * @throws InterruptedException if the thread is interrupted while waiting.
+ */
+ private long processCDCBatch(String partitionId, String ownerPartitionId,
+ long lastProcessedTimestamp, boolean isParentReplay)
+ throws SQLException, IOException, InterruptedException {
+ long batchStartTime = EnvironmentEdgeManager.currentTimeMillis();
+ LOG.debug("Processing CDC batch for table {} partition {} owner {} from timestamp {}",
+ dataTableName, partitionId, ownerPartitionId, lastProcessedTimestamp);
+ try (PhoenixConnection conn =
+ QueryUtil.getConnectionOnServer(config).unwrap(PhoenixConnection.class)) {
+ initTenantInfo(conn);
+ String cdcObjectName = getCdcObjectName(conn);
+ TenantScanInfo scanInfo = getPartitionTenantScanInfo(partitionId);
+ String cdcQuery;
+ if (isParentReplay) {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ } else {
+ cdcQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() > ? "
+ + "AND PHOENIX_ROW_TIMESTAMP() < ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC LIMIT ?",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ }
+ List<Pair<Long, IndexMutationsProtos.IndexMutations>> batchMutations = new ArrayList<>();
+ long newLastTimestamp = lastProcessedTimestamp;
+ boolean hasMoreRows = true;
+ int retryCount = 0;
+ while (hasMoreRows && batchMutations.isEmpty()) {
+ try (PreparedStatement ps = conn.prepareStatement(cdcQuery)) {
+ setStatementParams(scanInfo, partitionId, isParentReplay, newLastTimestamp, ps);
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ hasMoreRows = result.getSecond();
+ if (hasMoreRows) {
+ newLastTimestamp = result.getFirst();
+ if (batchMutations.isEmpty()) {
+ sleepIfNotStopped(ConnectionUtils.getPauseTime(pause, ++retryCount));
+ }
+ }
+ }
+ }
+ // With predefined LIMIT, there might be more rows with the same timestamp that were not
+ // included in this batch.
+ if (newLastTimestamp > lastProcessedTimestamp) {
+ String sameTimestampQuery = String.format(
+ "SELECT /*+ CDC_INCLUDE(IDX_MUTATIONS) */ PHOENIX_ROW_TIMESTAMP(), \"CDC JSON\" "
+ + "FROM %s WHERE %s PARTITION_ID() = ? AND PHOENIX_ROW_TIMESTAMP() = ? "
+ + "ORDER BY %s PARTITION_ID() ASC, PHOENIX_ROW_TIMESTAMP() ASC",
+ cdcObjectName, scanInfo.filter, scanInfo.orderBy);
+ final long timestampToRefetch = newLastTimestamp;
+ batchMutations.removeIf(pair -> pair.getFirst() == timestampToRefetch);
+ try (PreparedStatement ps = conn.prepareStatement(sameTimestampQuery)) {
+ int idx = scanInfo.bindParams(ps, 1);
+ ps.setString(idx++, partitionId);
+ ps.setDate(idx, new Date(newLastTimestamp));
+ Pair<Long, Boolean> result =
+ getMutationsAndTimestamp(ps, newLastTimestamp, batchMutations);
+ newLastTimestamp = result.getFirst();
+ if (newLastTimestamp != timestampToRefetch) {
+ throw new IOException("Unexpected timestamp mismatch: expected " + timestampToRefetch
+ + " but got " + newLastTimestamp);
+ }
+ }
+ }
+ executeIndexMutations(partitionId, batchMutations, ownerPartitionId, newLastTimestamp);
+ if (!batchMutations.isEmpty()) {
+ metricSource.updateCdcBatchProcessTime(dataTableName,
+ EnvironmentEdgeManager.currentTimeMillis() - batchStartTime);
+ metricSource.incrementCdcBatchCount(dataTableName);
+ updateTrackerProgress(conn, partitionId, ownerPartitionId, newLastTimestamp,
+ PhoenixDatabaseMetaData.TRACKER_STATUS_IN_PROGRESS);
+ }
+ return newLastTimestamp;
+ }
+ }
+
+ private String getCdcObjectName(PhoenixConnection conn) throws SQLException {
+ PTable dataTable = getDataTable(conn);
+ String cdcObjectName = CDCUtil.getCDCObjectName(dataTable, false);
+ if (cdcObjectName == null) {
+ throw new SQLException("No CDC object found for table " + dataTableName);
+ }
+ String schemaName = dataTable.getSchemaName().getString();
+ if (schemaName == null || schemaName.isEmpty()) {
+ cdcObjectName = "\"" + cdcObjectName + "\"";
+ } else {
+ cdcObjectName =
+ "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + cdcObjectName + "\"";
+ }
Review Comment:
```
public static String getEscapedTableName(String schemaName, String tableName) {
if (schemaName == null || schemaName.length() == 0) {
return "\"" + tableName + "\"";
}
return "\"" + schemaName + "\"" + QueryConstants.NAME_SEPARATOR + "\"" + tableName + "\"";
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org