You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "virajjasani (via GitHub)" <gi...@apache.org> on 2025/07/03 00:48:21 UTC
[PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
virajjasani opened a new pull request, #2209:
URL: https://github.com/apache/phoenix/pull/2209
Jira: PHOENIX-7653
The purpose of this PR is to extend the Change Data Capture (CDC) capabilities to generate CDC events when rows expire due to Time-To-Live (TTL) settings (literal or conditional) during the major compaction. The implementation ensures that applications consuming CDC streams receive notification when data is automatically removed from tables, providing additional visibility into the system-initiated deletions.
The proposed new event_type: ttl_delete
Example of TTL expired CDC event, assuming the row had two columns c1 and c2 with values "v1" and "v2" respectively:
```
{
"event_type": "ttl_delete",
"pre_image": {
"c1": "v1",
"c2": "v2"
},
"post_image": {}
}
```
High level Design steps:
- Identify the event which causes the row expiration: conditional_ttl, maxlookback/ttl expired rows
- Capture the complete row image for the expiration. The image needs to be directly inserted into the CDC index. If we do not provide the expired row pre-image upfront, CDC index can not scan it after the major compaction because the data table row no longer exists after it is expired by the major compaction. CompactionScanner needs to send the exact CDC Json structure with encoded bytes, which can later be directly sent to the client by the scanner when requested.
- CDCGlobalIndexRegionScanner needs to check for the existence of the special CF:CQ, which if found, can be directly returned as the value of "CDC JSON" column.
- For single CF, CompactionScanner needs to perform mutation to the CDC index directly only once.
- For multi CF, CompactionScanner might perform multiple mutation to the CDC index. Therefore, it should use checkAndMutate to ensure the mutation happens if the row does not exist. If the row is already inserted, and the other CF compaction tries to put recent row values, it can update the existing pre-image.
- In order to distinguish the same PHOENIX_ROW_TIMESTAMP() value for the CDC index while multiple CF compactions are taking place, CompactionScanner needs to provide compactionTime as the timestamp value in the CDC index rowkey by updating the rowkey before performing the mutation.
- Introduce some retries in case of HTable mutation failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188078842
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
Review Comment:
Sorry, i did not get it. You mean `change_image`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2189360525
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
Changes are read typically only once, but in any case, CDC is already known to have the overhead of building the image while reading and majority of the changes will come in through this path so why optimize just for a corner case scenario? Actually, if you compare the overhead against a regular delete, overhead for the new delete_ttl will anyway be less since the images would be partially built.
> read becomes simpler
Not sure what you mean, there would be no difference to the client and merging at the time of read just moves some of the already existing complexity from write to read, it shouldn't add anything new. Please let me know if I am missing something.
> we anyways cannot avoid CheckAndMutate
May be I don't fully understand this, but I think HBase already has deterministic logic around this where it gives priority to Delete over Put as well as use MVCC order so change change event can follow the same and be made deterministic, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190651383
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
These can be hidden CF, not really exposed to end users as they won't have any metadata in syscat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190068682
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
Interesting... I found that unless `NEW_VERSION_BEHAVIOR` CF attribute is set, HBase gives Delete a priority over Put in this case, irrespective of MVCC sequence number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190116124
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -259,4 +310,44 @@ private Object getColumnValue(byte[] cellValue, int offset, int length, PDataTyp
}
return CDCUtil.getColumnEncodedValue(value, dataType);
}
+
+ /**
+ * Handles CDC events that already contain pre-image data, avoiding data table scan.
+ * Supports both the new CDC_IMAGE_CQ column and traditional CDC JSON column.
+ *
+ * @param indexRow The CDC index row cells
+ * @param indexRowKey The CDC index row key
+ * @param indexCell The primary index cell
+ * @param result The result list to populate
+ * @return true if event was processed successfully
+ */
+ private boolean handlePreImageCDCEvent(List<Cell> indexRow, byte[] indexRowKey,
+ Cell indexCell, List<Cell> result) {
+ Cell cdcDataCell = null;
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ cdcDataCell = cell;
+ break;
+ }
+ }
+ if (cdcDataCell == null) {
+ return false;
+ }
+ byte[] cdcEventBytes = CellUtil.cloneValue(cdcDataCell);
Review Comment:
The if condition is meant to avoid creating garbage when not required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190984383
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
If indexes are made multi-CF, their compaction can also become multi-CF. We do not support multi-CF in Index table by design. Adding new CF is table level DDL operation, we cannot know in advance what CF needs to be added in the uncovered index table.
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
We do not support multi-CF in Index table by design. Adding new CF is table level DDL operation, we cannot know in advance what CF needs to be added in the uncovered index table. That's the reason even covered index does not support multi-CF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190482509
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -259,4 +310,44 @@ private Object getColumnValue(byte[] cellValue, int offset, int length, PDataTyp
}
return CDCUtil.getColumnEncodedValue(value, dataType);
}
+
+ /**
+ * Handles CDC events that already contain pre-image data, avoiding data table scan.
+ * Supports both the new CDC_IMAGE_CQ column and traditional CDC JSON column.
+ *
+ * @param indexRow The CDC index row cells
+ * @param indexRowKey The CDC index row key
+ * @param indexCell The primary index cell
+ * @param result The result list to populate
+ * @return true if event was processed successfully
+ */
+ private boolean handlePreImageCDCEvent(List<Cell> indexRow, byte[] indexRowKey,
+ Cell indexCell, List<Cell> result) {
+ Cell cdcDataCell = null;
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ cdcDataCell = cell;
+ break;
+ }
+ }
+ if (cdcDataCell == null) {
+ return false;
+ }
+ byte[] cdcEventBytes = CellUtil.cloneValue(cdcDataCell);
Review Comment:
Yes exactly, the optimization will not be applicable here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2191000315
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
Also, given the increasing complexity of CompactionScanner, we need to keep the workflow as simple as possible. In the worst case, if the mutation to the CDC index fails after configured attempts, we still need to not block the compaction because the cost could be very high depending on the size of the rows, tables, regions and stores.
That's another reason i wanted to keep the workflow as simple as:
- Generate mutation and perform CheckAndMutate (success for single CF case)
- If CheckAndMutate fails, merge the image (multi-CF case, simple enough because the latest mutation will be retrieved from the region memstore quickly)
- If CheckAndMutate fails due to UPSERT event right at the exact timestamp, ignore CDC event for TTL because even though TTL is expired, the new UPSERT happened at the exact moment so upsert event makes precedence (extreme rare event)
I found the above logic at one place (because TTL_DELETE is special event) in static utility more manageable than making changes to CDCTableInfo, and creating new objects of CDCTableInfo and CDCChangeBuilder for each TTL expired row. In current case, we are not even creating new object of CDCCompactionUtil.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2191000315
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
Also, given the increasing complexity of CompactionScanner, we need to keep the workflow as simple as possible. In the worst case, if the mutation to the CDC index fails after configured attempts, we still need to not block the compaction because the cost could be very high depending on the size of the rows, tables, regions and stores.
That's another reason i wanted to keep the workflow as simple as:
- Generate mutation and perform CheckAndMutate (success for single CF case)
- If CheckAndMutate fails, merge the image (multi-CF case, simple enough because the latest mutation will be retrieved from the region memstore quickly)
- If CheckAndMutate fails due to UPSERT event right at the exact timestamp, ignore CDC event for TTL because even though TTL is expired, the new UPSERT happened at the exact moment so upsert event makes precedence
I found the above logic at one place (because TTL_DELETE is special event) in static utility more manageable than making changes to CDCTableInfo, and creating new objects of CDCTableInfo and CDCChangeBuilder for each TTL expired row. In current case, we are not even creating new object of CDCCompactionUtil.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188088722
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
Also, we anyways cannot avoid CheckAndMutate operation because of the possibility of rare event (for which i have provided warning log) where some change event (upsert) happened to the exact row at the exact time when CompactionScanner was trying to put TTL delete event in the CDC index.
Therefore since we need CheckAndMutate atomic operation, we better also use it to further merge the row state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188082200
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
Also, even if we could initialize CDCTableInfo directly, we can still not use `CDCChangeBuilder` because the object used by `CDCGlobalIndexRegionScanner` is different from the one `CompactionScanner` would use. So, let's say even if `CompactionScanner` could register change, it cannot be detected by `CDCGlobalIndexRegionScanner` because both of these would be different objects.
It is only as part of `getNextCoveredIndexRow()` that `CDCGlobalIndexRegionScanner` can register the change based on the data scanned from the data table, and use the same object to build the pre-image and post-image for `CDC JSON` column cell dynamically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190511410
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
Review Comment:
Makes sense, although it's interesting that we can only provide PRE scope for TLL_DELETE event (similar to DELETE event) so we anyways can return only pre-image. For instance, someone using `CDC_INCLUDE(CHANGE,POST)` will not need anything to return for this case.
If you don't mind, let me create follow-up Jira to fix the scope, given that the PR is already big enough for the feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190984383
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
If indexes are made multi-CF, their compaction can also become multi-CF. We do not support multi-CF in Index table by design. Adding new CF is table level DDL operation, we cannot know in advance what CF needs to be added in the uncovered index table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2189321154
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
Review Comment:
I meant honouring the default scopes specified at the time of creating CDC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #2209:
URL: https://github.com/apache/phoenix/pull/2209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188078156
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
This was the initial idea we discussed but providing the in-built image seems far efficient because the same change event can be consumed multiple times (by multiple consumers). When that happens, we will keep deserializing cells on the fly, rather than let the compaction take care of deserializing to the exact bytes that the scanner will return without any additional logic. This approach seemed more efficient in the case where we need to provide pre-built image.
In other normal cases (upsert and delete), since we are scanning the data table, there is anyways no scope for bringing extra efficiency in compute while scanning and forming "CDC JSON" column cell. However, here we can make it much simpler: let compaction write the change image, whereas scanner can read it multiple times without any additional compute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188069520
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -259,4 +310,44 @@ private Object getColumnValue(byte[] cellValue, int offset, int length, PDataTyp
}
return CDCUtil.getColumnEncodedValue(value, dataType);
}
+
+ /**
+ * Handles CDC events that already contain pre-image data, avoiding data table scan.
+ * Supports both the new CDC_IMAGE_CQ column and traditional CDC JSON column.
+ *
+ * @param indexRow The CDC index row cells
+ * @param indexRowKey The CDC index row key
+ * @param indexCell The primary index cell
+ * @param result The result list to populate
+ * @return true if event was processed successfully
+ */
+ private boolean handlePreImageCDCEvent(List<Cell> indexRow, byte[] indexRowKey,
+ Cell indexCell, List<Cell> result) {
+ Cell cdcDataCell = null;
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ cdcDataCell = cell;
+ break;
+ }
+ }
+ if (cdcDataCell == null) {
+ return false;
+ }
+ byte[] cdcEventBytes = CellUtil.cloneValue(cdcDataCell);
Review Comment:
To retrieve value bytes directly from Cell object, we use `CellUtil.cloneValue()` almost every place possible.
On the other hand `copyBytesIfNecessary()` is also doing copy similar to `CellUtil.cloneValue()`:
```
public static byte[] copyBytesIfNecessary(byte[] bytes, int offset, int length) {
if (offset == 0 && length == bytes.length) {
return bytes;
}
return Arrays.copyOfRange(bytes, offset, offset + length);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188082200
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
Also, even if we could initialize CDCTableInfo directly, we can still not use `CDCChangeBuilder` because the object used by `CDCGlobalIndexRegionScanner` is different from the one `CompactionScanner` would use. So, let's say even if `CompactionScanner` could register change, it cannot be detected by `CDCGlobalIndexRegionScanner` because both of these would be different objects.
It is only as part of `getNextCoveredIndexRow()` that `CDCGlobalIndexRegionScanner` can register the change based on the data scanned from the data table, and use the same object to build the pre-image and post-image for `CDC JSON` column cell dynamically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190128056
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
I agree existing code is not directly usable as it doesn't expose fine grained utilities, but interestingly, this is what you did so we could probably refactor the existing code to use your new code and reduce some duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2183376694
##########
phoenix-core-client/src/main/java/org/apache/phoenix/util/CDCUtil.java:
##########
@@ -152,7 +171,7 @@ public static int compareCellFamilyAndQualifier(byte[] columnFamily1,
public static Object getColumnEncodedValue(Object value, PDataType dataType) {
if (value != null) {
if (dataType.getSqlType() == PDataType.BSON_TYPE) {
- value = Bytes.toBytes(((RawBsonDocument) value).getByteBuffer().asNIO());
+ value = ByteUtil.toBytes(((RawBsonDocument) value).getByteBuffer().asNIO());
Review Comment:
In addition to https://github.com/apache/phoenix/pull/2162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190475589
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
We are only dealing with Put mutations for the index table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2191489483
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
Review Comment:
FYI @haridsv, created the follow-up Jira PHOENIX-7658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190987096
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
We do not support multi-CF in Index table by design. Adding new CF is table level DDL operation, we cannot know in advance what CF needs to be added in the uncovered index table. That's the reason even covered index does not support multi-CF.
Adding CF will require HBase Admin's modifyTableDescriptor() API, followed by region restarts:
```
/**
* Modify an existing table, more IRB friendly version.
* @param td modified description of the table
* @throws IOException if a remote or network exception occurs
*/
default void modifyTable(TableDescriptor td) throws IOException {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2183909844
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null ?
+ dataTable.getDefaultFamilyName().getBytes() :
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null &&
+ Bytes.equals(family, column.getFamilyName().getBytes()) &&
+ Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount <= cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
+
+ cdcIndexTable.put(mergedCdcIndexPut);
+ lastException = null;
+ break;
+ } else {
+ LOGGER.warn("Rare event: Skipping CDC TTL mutation because other type"
+ + " of CDC event is recorded at time {}", eventTimestamp);
+ }
+ }
+ }
+ } catch (Exception e) {
+ lastException = e;
+ if (retryCount < cdcTtlMutationMaxRetries) {
+ long backoffMs = 100;
+ LOGGER.warn("CDC mutation attempt {}/{} failed, retrying in {}ms",
+ retryCount + 1, cdcTtlMutationMaxRetries + 1, backoffMs, e);
+ try {
+ Thread.sleep(backoffMs);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ throw new IOException("Interrupted during CDC mutation retry", ie);
+ }
+ }
+ }
+ }
+ if (lastException != null) {
+ LOGGER.error("Failed to generate CDC mutation after {} attempts for table {}, index " +
+ "{}. The event update is missed.",
+ cdcTtlMutationMaxRetries + 1,
+ tableName,
+ cdcIndex.getPhysicalName().getString(),
+ lastException);
+ }
+ }
+
+ /**
+ * Generates CDC TTL delete event and writes it directly to CDC index tables.
+ * This bypasses the normal CDC update path since the row is being expired.
+ *
+ * @param expiredRow The cells of the expired row
+ * @param tableName The table name for logging
+ * @param compactionTime The compaction timestamp
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCTTLDeleteEvent(List<Cell> expiredRow, String tableName,
+ long compactionTime, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries) {
+ if (expiredRow.isEmpty()) {
+ return;
+ }
+ try {
+ List<PTable> cdcIndexes = new ArrayList<>();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2183909346
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CompactionScanner.java:
##########
@@ -205,7 +212,15 @@ public CompactionScanner(RegionCoprocessorEnvironment env,
emptyCFStore = familyCount == 1 || columnFamilyName.equals(Bytes.toString(emptyCF))
|| localIndex;
- isCDCIndex = table != null ? CDCUtil.isCDCIndex(table) : false;
+ this.table = table;
+ isCDCIndex = CDCUtil.isCDCIndex(table);
Review Comment:
`isCDCIndex` is already in use, this change is just optimization from `isCDCIndex = table != null ? CDCUtil.isCDCIndex(table) : false` to `isCDCIndex = CDCUtil.isCDCIndex(table)`
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -259,4 +307,71 @@ private Object getColumnValue(byte[] cellValue, int offset, int length, PDataTyp
}
return CDCUtil.getColumnEncodedValue(value, dataType);
}
+
+ /**
+ * Checks if the CDC index row already contains pre-image data.
+ * This is true for TTL delete events that embed complete row data.
+ * Checks for the new CDC_IMAGE_CQ column that contains pre-computed CDC event data.
+ *
+ * @param indexRow The CDC index row cells
+ * @return true if pre-image data is available, false if data table scan is needed
+ */
+ private boolean hasPreImageData(List<Cell> indexRow) {
+ try {
+ if (indexRow.size() == 1) {
+ return false;
+ }
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ return true;
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error checking for pre-image data in CDC index row", e);
+ }
+ return false;
+ }
+
+ /**
+ * Handles CDC events that already contain pre-image data, avoiding data table scan.
+ * Supports both the new CDC_IMAGE_CQ column and traditional CDC JSON column.
+ *
+ * @param indexRow The CDC index row cells
+ * @param indexRowKey The CDC index row key
+ * @param indexCell The primary index cell
+ * @param result The result list to populate
+ * @return true if event was processed successfully
+ */
+ private boolean handlePreImageCDCEvent(List<Cell> indexRow, byte[] indexRowKey,
+ Cell indexCell, List<Cell> result) {
+ Cell cdcDataCell = null;
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ cdcDataCell = cell;
+ break;
+ }
+ }
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2183909689
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null ?
+ dataTable.getDefaultFamilyName().getBytes() :
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null &&
+ Bytes.equals(family, column.getFamilyName().getBytes()) &&
+ Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount <= cdcTtlMutationMaxRetries; retryCount++) {
Review Comment:
done
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null ?
+ dataTable.getDefaultFamilyName().getBytes() :
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null &&
+ Bytes.equals(family, column.getFamilyName().getBytes()) &&
+ Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount <= cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
+
+ cdcIndexTable.put(mergedCdcIndexPut);
+ lastException = null;
+ break;
+ } else {
+ LOGGER.warn("Rare event: Skipping CDC TTL mutation because other type"
+ + " of CDC event is recorded at time {}", eventTimestamp);
+ }
+ }
+ }
+ } catch (Exception e) {
+ lastException = e;
+ if (retryCount < cdcTtlMutationMaxRetries) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2183908884
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CompactionScanner.java:
##########
@@ -205,7 +212,15 @@ public CompactionScanner(RegionCoprocessorEnvironment env,
emptyCFStore = familyCount == 1 || columnFamilyName.equals(Bytes.toString(emptyCF))
|| localIndex;
- isCDCIndex = table != null ? CDCUtil.isCDCIndex(table) : false;
+ this.table = table;
+ isCDCIndex = CDCUtil.isCDCIndex(table);
Review Comment:
It is used to ensure the cells of CDC index that are in the maxLookback window are always retained
```
if (isCDCIndex) {
return retainCellsForCDCIndex(result, retainedCells);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2183945222
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CompactionScanner.java:
##########
@@ -205,7 +212,15 @@ public CompactionScanner(RegionCoprocessorEnvironment env,
emptyCFStore = familyCount == 1 || columnFamilyName.equals(Bytes.toString(emptyCF))
|| localIndex;
- isCDCIndex = table != null ? CDCUtil.isCDCIndex(table) : false;
+ this.table = table;
+ isCDCIndex = CDCUtil.isCDCIndex(table);
Review Comment:
got it, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2185482029
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -259,4 +310,44 @@ private Object getColumnValue(byte[] cellValue, int offset, int length, PDataTyp
}
return CDCUtil.getColumnEncodedValue(value, dataType);
}
+
+ /**
+ * Handles CDC events that already contain pre-image data, avoiding data table scan.
+ * Supports both the new CDC_IMAGE_CQ column and traditional CDC JSON column.
+ *
+ * @param indexRow The CDC index row cells
+ * @param indexRowKey The CDC index row key
+ * @param indexCell The primary index cell
+ * @param result The result list to populate
+ * @return true if event was processed successfully
+ */
+ private boolean handlePreImageCDCEvent(List<Cell> indexRow, byte[] indexRowKey,
+ Cell indexCell, List<Cell> result) {
+ Cell cdcDataCell = null;
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ cdcDataCell = cell;
+ break;
+ }
+ }
+ if (cdcDataCell == null) {
+ return false;
+ }
+ byte[] cdcEventBytes = CellUtil.cloneValue(cdcDataCell);
Review Comment:
Can we not make use of the optimization done by `ImmutableBytesPtr.cloneCellValueIfNecessary`?
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
Can this not build CDCTableInfo and use it with CDCChangeBuilder to avoid some duplication of logic?
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
This logic to incrementally merge would be quite inefficient right? What if we use a separate CF in Index, one for each CF in data table and merge them on read?
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
Review Comment:
What about CDC scope?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#issuecomment-3041055668
> Have you looked into a way to reuse the existing logic for building change images instead of building a parallel logic in the new CDCCompactionUtil class?
Yes i did but there are 2 concerns:
1. CDCTableInfo cannot be directly constructed from data table PTable. Currently, it only provides a way to get initialized using serialization and deserialization path. This makes it difficult to directly use it at the server side.
2. Even if CDCTableInfo and CDCColumnInfo could be used directly at the server side, they provide a redundant layer on top of what can already be constructed with minimal/simpler efforts i.e. mutation for uncovered index with pre-built image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2183863160
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null ?
+ dataTable.getDefaultFamilyName().getBytes() :
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null &&
+ Bytes.equals(family, column.getFamilyName().getBytes()) &&
+ Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount <= cdcTtlMutationMaxRetries; retryCount++) {
Review Comment:
`retryCount < cdcTtlMutationMaxRetries`?
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null ?
+ dataTable.getDefaultFamilyName().getBytes() :
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null &&
+ Bytes.equals(family, column.getFamilyName().getBytes()) &&
+ Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount <= cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
+
+ cdcIndexTable.put(mergedCdcIndexPut);
+ lastException = null;
+ break;
+ } else {
+ LOGGER.warn("Rare event: Skipping CDC TTL mutation because other type"
+ + " of CDC event is recorded at time {}", eventTimestamp);
+ }
+ }
+ }
+ } catch (Exception e) {
+ lastException = e;
+ if (retryCount < cdcTtlMutationMaxRetries) {
+ long backoffMs = 100;
+ LOGGER.warn("CDC mutation attempt {}/{} failed, retrying in {}ms",
+ retryCount + 1, cdcTtlMutationMaxRetries + 1, backoffMs, e);
+ try {
+ Thread.sleep(backoffMs);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ throw new IOException("Interrupted during CDC mutation retry", ie);
+ }
+ }
+ }
+ }
+ if (lastException != null) {
+ LOGGER.error("Failed to generate CDC mutation after {} attempts for table {}, index " +
+ "{}. The event update is missed.",
+ cdcTtlMutationMaxRetries + 1,
+ tableName,
+ cdcIndex.getPhysicalName().getString(),
+ lastException);
+ }
+ }
+
+ /**
+ * Generates CDC TTL delete event and writes it directly to CDC index tables.
+ * This bypasses the normal CDC update path since the row is being expired.
+ *
+ * @param expiredRow The cells of the expired row
+ * @param tableName The table name for logging
+ * @param compactionTime The compaction timestamp
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCTTLDeleteEvent(List<Cell> expiredRow, String tableName,
+ long compactionTime, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries) {
+ if (expiredRow.isEmpty()) {
+ return;
+ }
+ try {
+ List<PTable> cdcIndexes = new ArrayList<>();
Review Comment:
We only allow one CDC now since the Stream changes so this can be simplified.
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CompactionScanner.java:
##########
@@ -205,7 +212,15 @@ public CompactionScanner(RegionCoprocessorEnvironment env,
emptyCFStore = familyCount == 1 || columnFamilyName.equals(Bytes.toString(emptyCF))
|| localIndex;
- isCDCIndex = table != null ? CDCUtil.isCDCIndex(table) : false;
+ this.table = table;
+ isCDCIndex = CDCUtil.isCDCIndex(table);
Review Comment:
where is `isCDCIndex` being used?
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -259,4 +307,71 @@ private Object getColumnValue(byte[] cellValue, int offset, int length, PDataTyp
}
return CDCUtil.getColumnEncodedValue(value, dataType);
}
+
+ /**
+ * Checks if the CDC index row already contains pre-image data.
+ * This is true for TTL delete events that embed complete row data.
+ * Checks for the new CDC_IMAGE_CQ column that contains pre-computed CDC event data.
+ *
+ * @param indexRow The CDC index row cells
+ * @return true if pre-image data is available, false if data table scan is needed
+ */
+ private boolean hasPreImageData(List<Cell> indexRow) {
+ try {
+ if (indexRow.size() == 1) {
+ return false;
+ }
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ return true;
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error checking for pre-image data in CDC index row", e);
+ }
+ return false;
+ }
+
+ /**
+ * Handles CDC events that already contain pre-image data, avoiding data table scan.
+ * Supports both the new CDC_IMAGE_CQ column and traditional CDC JSON column.
+ *
+ * @param indexRow The CDC index row cells
+ * @param indexRowKey The CDC index row key
+ * @param indexCell The primary index cell
+ * @param result The result list to populate
+ * @return true if event was processed successfully
+ */
+ private boolean handlePreImageCDCEvent(List<Cell> indexRow, byte[] indexRowKey,
+ Cell indexCell, List<Cell> result) {
+ Cell cdcDataCell = null;
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ cdcDataCell = cell;
+ break;
+ }
+ }
Review Comment:
`hasPreImageData` can return the cell if found so that we can avoid searching for it again here.
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null ?
+ dataTable.getDefaultFamilyName().getBytes() :
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null &&
+ Bytes.equals(family, column.getFamilyName().getBytes()) &&
+ Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount <= cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
+
+ cdcIndexTable.put(mergedCdcIndexPut);
+ lastException = null;
+ break;
+ } else {
+ LOGGER.warn("Rare event: Skipping CDC TTL mutation because other type"
+ + " of CDC event is recorded at time {}", eventTimestamp);
+ }
+ }
+ }
+ } catch (Exception e) {
+ lastException = e;
+ if (retryCount < cdcTtlMutationMaxRetries) {
Review Comment:
can be removed since we are already in a retry loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188078156
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
This was the initial idea we discussed but providing the in-built image seems far efficient because the same change event can be consumed multiple times (by multiple consumers). When that happens, we will keep deserializing cells on the fly, rather than let the compaction take care of deserializing to the exact bytes that the scanner will return without any additional logic. This approach seemed more efficient in the case where we need to provide pre-built image.
In other normal cases (upsert and delete), since we are scanning the data table, there is anyways no scope for bringing extra scope efficiency. However, here we can make it much simpler: let compaction write the change image, whereas scanner can read it multiple times without any additional compute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188082200
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
Also, even if we could initialize CDCTableInfo directly, we can still not use `CDCChangeBuilder` because the object used by `CDCGlobalIndexRegionScanner` is different from the one `CompactionScanner` uses. So, let's say even if `CompactionScanner` could register change, it cannot be detected by `CDCGlobalIndexRegionScanner` because both of these would be different objects.
It is only as part of `getNextCoveredIndexRow()` that `CDCGlobalIndexRegionScanner` can register the change based on the data scanned from the data table, and use the same object to build the pre-image and post-image for `CDC JSON` column cell dynamically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190480657
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
I tried that also, but did not find it straightforward, that's why I had to make it this way.
Btw it's completely fine if existing classes can be made simplified in future, then we can make both use same utilities, but I still think that route is complex for TTL_DELETE case because we don't need utilities (that we use while reading from the index table) while directly writing the built-in image to the index table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190116124
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -259,4 +310,44 @@ private Object getColumnValue(byte[] cellValue, int offset, int length, PDataTyp
}
return CDCUtil.getColumnEncodedValue(value, dataType);
}
+
+ /**
+ * Handles CDC events that already contain pre-image data, avoiding data table scan.
+ * Supports both the new CDC_IMAGE_CQ column and traditional CDC JSON column.
+ *
+ * @param indexRow The CDC index row cells
+ * @param indexRowKey The CDC index row key
+ * @param indexCell The primary index cell
+ * @param result The result list to populate
+ * @return true if event was processed successfully
+ */
+ private boolean handlePreImageCDCEvent(List<Cell> indexRow, byte[] indexRowKey,
+ Cell indexCell, List<Cell> result) {
+ Cell cdcDataCell = null;
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ cdcDataCell = cell;
+ break;
+ }
+ }
+ if (cdcDataCell == null) {
+ return false;
+ }
+ byte[] cdcEventBytes = CellUtil.cloneValue(cdcDataCell);
Review Comment:
The if condition is meant to reduce garbage when possible. The other one always clones the byte array. Of course this shouldn't be used when the byte array needs to be kept around, but I think it is not the case here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190520567
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
>> read becomes simpler
>
> Not sure what you mean, there would be no difference to the client and merging at the time of read just moves some of the already existing complexity from write to read, it shouldn't add anything new. Please let me know if I am missing something.
The problem is we can't make CDC index as multi-CF index. Index tables by design are only single CF tables, both covered and uncovered IIRC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188072051
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
Review Comment:
By looking at how CDCTableInfo is built, there is no direct API to construct it from data table PTable object such that we don't have to go through serialization and deserialization path. That's why i did not pursue using CDCTableInfo. Also, CDCChangeBuilder is tightly coupled with CDCTableInfo and therefore only used by the scanner init:
```
CDCUtil.setupScanForCDC(dataTableScan);
cdcDataTableInfo = CDCTableInfo.createFromProto(CDCInfoProtos.CDCTableDef
.parseFrom(scan.getAttribute(CDC_DATA_TABLE_DEF)));
changeBuilder = new CDCChangeBuilder(cdcDataTableInfo);
```
Whereas here, we have much simple case: embed pre-built image with the index mutation because data table scan will not get us any data (major compacted).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188080271
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
Review Comment:
If you meant `event_type`, it will be `ttl_delete`:
```
cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2188078156
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
This was the initial idea we discussed but providing the in-built image seems far more efficient because the same change event can be consumed multiple times (by multiple consumers). When that happens, we will keep deserializing cells on the fly, rather than let the compaction take care of deserializing to the exact bytes that the scanner will return without any additional logic.
This approach seemed more efficient in the case where we need to provide pre-built image.
In other normal cases (upsert and delete), since we are scanning the data table, there is anyways no scope for bringing extra efficiency in compute while scanning and forming "CDC JSON" column cell. However, here we can make it much simpler: let compaction write the change image, whereas scanner can read it multiple times without any additional compute.
To sum up, when we need to scan the data table, we anyways deal with multi-CF. However, when we need to provide pre-built image, using single CF (default CF) is better so that read becomes simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190482509
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -259,4 +310,44 @@ private Object getColumnValue(byte[] cellValue, int offset, int length, PDataTyp
}
return CDCUtil.getColumnEncodedValue(value, dataType);
}
+
+ /**
+ * Handles CDC events that already contain pre-image data, avoiding data table scan.
+ * Supports both the new CDC_IMAGE_CQ column and traditional CDC JSON column.
+ *
+ * @param indexRow The CDC index row cells
+ * @param indexRowKey The CDC index row key
+ * @param indexCell The primary index cell
+ * @param result The result list to populate
+ * @return true if event was processed successfully
+ */
+ private boolean handlePreImageCDCEvent(List<Cell> indexRow, byte[] indexRowKey,
+ Cell indexCell, List<Cell> result) {
+ Cell cdcDataCell = null;
+ for (Cell cell : indexRow) {
+ if (Bytes.equals(cell.getQualifierArray(), cell.getQualifierOffset(),
+ cell.getQualifierLength(),
+ QueryConstants.CDC_IMAGE_CQ_BYTES, 0,
+ QueryConstants.CDC_IMAGE_CQ_BYTES.length)) {
+ cdcDataCell = cell;
+ break;
+ }
+ }
+ if (cdcDataCell == null) {
+ return false;
+ }
+ byte[] cdcEventBytes = CellUtil.cloneValue(cdcDataCell);
Review Comment:
Yes exactly, the optimization will not be applicable here because neither of the conditions will be true for Cell value[].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#discussion_r2190987096
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/CDCCompactionUtil.java:
##########
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.expression.function.PartitionIdFunction;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PDate;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.JacksonUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.query.QueryConstants.NAME_SEPARATOR;
+
+/**
+ * Utility class for CDC (Change Data Capture) operations during compaction.
+ * This class contains utilities for handling TTL row expiration events and generating
+ * CDC events with pre-image data that are written directly to CDC index tables.
+ */
+public final class CDCCompactionUtil {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(CDCCompactionUtil.class);
+
+ private CDCCompactionUtil() {
+ // empty
+ }
+
+ /**
+ * Finds the column name for a given cell in the data table.
+ *
+ * @param dataTable The data table
+ * @param cell The cell
+ * @return The column name or null if not found
+ */
+ static String findColumnName(PTable dataTable, Cell cell) {
+ try {
+ byte[] family = CellUtil.cloneFamily(cell);
+ byte[] qualifier = CellUtil.cloneQualifier(cell);
+ byte[] defaultCf = dataTable.getDefaultFamilyName() != null
+ ? dataTable.getDefaultFamilyName().getBytes()
+ : QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES;
+ for (PColumn column : dataTable.getColumns()) {
+ if (column.getFamilyName() != null
+ && Bytes.equals(family, column.getFamilyName().getBytes())
+ && Bytes.equals(qualifier, column.getColumnQualifierBytes())) {
+ if (Bytes.equals(defaultCf, column.getFamilyName().getBytes())) {
+ return column.getName().getString();
+ } else {
+ return column.getFamilyName().getString() + NAME_SEPARATOR
+ + column.getName().getString();
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error finding column name for cell: {}", CellUtil.toString(cell, true),
+ e);
+ }
+ return null;
+ }
+
+ /**
+ * Creates a CDC event map for TTL delete with pre-image data.
+ *
+ * @param expiredRowPut The expired row data
+ * @param dataTable The data table
+ * @param preImage Pre-image map
+ * @return CDC event map
+ */
+ static Map<String, Object> createTTLDeleteCDCEvent(Put expiredRowPut, PTable dataTable,
+ Map<String, Object> preImage)
+ throws Exception {
+ Map<String, Object> cdcEvent = new HashMap<>();
+ cdcEvent.put(QueryConstants.CDC_EVENT_TYPE, QueryConstants.CDC_TTL_DELETE_EVENT_TYPE);
+ for (List<Cell> familyCells : expiredRowPut.getFamilyCellMap().values()) {
+ for (Cell cell : familyCells) {
+ String columnName = findColumnName(dataTable, cell);
+ if (columnName != null) {
+ PColumn column = dataTable.getColumnForColumnQualifier(
+ CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell));
+ Object value = column.getDataType().toObject(cell.getValueArray(),
+ cell.getValueOffset(),
+ cell.getValueLength());
+ Object encodedValue =
+ CDCUtil.getColumnEncodedValue(value, column.getDataType());
+ preImage.put(columnName, encodedValue);
+ }
+ }
+ }
+ cdcEvent.put(QueryConstants.CDC_PRE_IMAGE, preImage);
+ cdcEvent.put(QueryConstants.CDC_POST_IMAGE, Collections.emptyMap());
+ return cdcEvent;
+ }
+
+ /**
+ * Builds CDC index Put mutation.
+ *
+ * @param cdcIndex The CDC index table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param cdcEventBytes The CDC event data to store
+ * @param dataTable The data table
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @return The CDC index Put mutation
+ */
+ static Put buildCDCIndexPut(PTable cdcIndex, Put expiredRowPut, long eventTimestamp,
+ byte[] cdcEventBytes, PTable dataTable,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes) throws Exception {
+
+ try (PhoenixConnection serverConnection = QueryUtil.getConnectionOnServer(new Properties(),
+ env.getConfiguration()).unwrap(PhoenixConnection.class)) {
+
+ IndexMaintainer cdcIndexMaintainer =
+ cdcIndex.getIndexMaintainer(dataTable, serverConnection);
+
+ ValueGetter dataRowVG = new IndexUtil.SimpleValueGetter(expiredRowPut);
+ ImmutableBytesPtr rowKeyPtr = new ImmutableBytesPtr(expiredRowPut.getRow());
+
+ Put cdcIndexPut = cdcIndexMaintainer.buildUpdateMutation(
+ GenericKeyValueBuilder.INSTANCE,
+ dataRowVG,
+ rowKeyPtr,
+ eventTimestamp,
+ null,
+ null,
+ false,
+ region.getRegionInfo().getEncodedNameAsBytes());
+
+ byte[] rowKey = cdcIndexPut.getRow().clone();
+ System.arraycopy(compactionTimeBytes, 0, rowKey,
+ PartitionIdFunction.PARTITION_ID_LENGTH, PDate.INSTANCE.getByteSize());
+ Put newCdcIndexPut = new Put(rowKey, eventTimestamp);
+
+ newCdcIndexPut.addColumn(
+ cdcIndexMaintainer.getEmptyKeyValueFamily().copyBytesIfNecessary(),
+ cdcIndexMaintainer.getEmptyKeyValueQualifier(), eventTimestamp,
+ QueryConstants.UNVERIFIED_BYTES);
+
+ // Add CDC event data
+ newCdcIndexPut.addColumn(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES,
+ eventTimestamp,
+ cdcEventBytes);
+
+ return newCdcIndexPut;
+ }
+ }
+
+ /**
+ * Generates and applies a CDC index mutation for TTL expired row with retries if required.
+ *
+ * @param cdcIndex The CDC index table
+ * @param dataTable The data table
+ * @param expiredRowPut The expired row data as a Put
+ * @param eventTimestamp The timestamp for the CDC event
+ * @param tableName The table name for logging
+ * @param env The region coprocessor environment
+ * @param region The HBase region
+ * @param compactionTimeBytes The compaction time as bytes
+ * @param cdcTtlMutationMaxRetries Maximum retry attempts for CDC mutations
+ */
+ static void generateCDCIndexMutation(PTable cdcIndex, PTable dataTable,
+ Put expiredRowPut,
+ long eventTimestamp, String tableName,
+ RegionCoprocessorEnvironment env, Region region,
+ byte[] compactionTimeBytes,
+ int cdcTtlMutationMaxRetries)
+ throws Exception {
+ Map<String, Object> cdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable, new HashMap<>());
+ byte[] cdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class).writeValueAsBytes(cdcEvent);
+ Put cdcIndexPut =
+ buildCDCIndexPut(cdcIndex, expiredRowPut, eventTimestamp, cdcEventBytes,
+ dataTable, env, region, compactionTimeBytes);
+
+ Exception lastException = null;
+ for (int retryCount = 0; retryCount < cdcTtlMutationMaxRetries; retryCount++) {
+ try (Table cdcIndexTable = env.getConnection().getTable(TableName.valueOf(
+ cdcIndex.getPhysicalName().getBytes()))) {
+ CheckAndMutate checkAndMutate =
+ CheckAndMutate.newBuilder(cdcIndexPut.getRow())
+ .ifNotExists(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES)
+ .build(cdcIndexPut);
+ CheckAndMutateResult result = cdcIndexTable.checkAndMutate(checkAndMutate);
+
+ if (result.isSuccess()) {
+ // Successfully inserted new CDC event - Single CF case
+ lastException = null;
+ break;
+ } else {
+ // Row already exists, need to retrieve existing pre-image and merge
+ // Likely to happen for multi CF case
+ Get get = new Get(cdcIndexPut.getRow());
+ get.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+ Result existingResult = cdcIndexTable.get(get);
+
+ if (!existingResult.isEmpty()) {
+ Cell existingCell = existingResult.getColumnLatestCell(
+ QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.CDC_IMAGE_CQ_BYTES);
+
+ if (existingCell != null) {
+ byte[] existingCdcBytes = CellUtil.cloneValue(existingCell);
+ Map<String, Object> existingCdcEvent =
+ JacksonUtil.getObjectReader(HashMap.class)
+ .readValue(existingCdcBytes);
+ Map<String, Object> existingPreImage =
+ (Map<String, Object>) existingCdcEvent.getOrDefault(
+ QueryConstants.CDC_PRE_IMAGE, new HashMap<>());
+
+ // Create new TTL delete event with merged pre-image
+ Map<String, Object> mergedCdcEvent =
+ createTTLDeleteCDCEvent(expiredRowPut, dataTable,
+ existingPreImage);
+ byte[] mergedCdcEventBytes =
+ JacksonUtil.getObjectWriter(HashMap.class)
+ .writeValueAsBytes(mergedCdcEvent);
+
+ Put mergedCdcIndexPut = buildCDCIndexPut(cdcIndex, expiredRowPut,
+ eventTimestamp, mergedCdcEventBytes, dataTable, env, region,
+ compactionTimeBytes);
Review Comment:
We do not support multi-CF in Index table by design. Adding new CF is table level DDL operation, we cannot know in advance what CF needs to be added in the uncovered index table. That's the reason even covered index does not support multi-CF.
Adding CF will require HBase Admin's modifyTableDescriptor() API:
```
/**
* Modify an existing table, more IRB friendly version.
* @param td modified description of the table
* @throws IOException if a remote or network exception occurs
*/
default void modifyTable(TableDescriptor td) throws IOException {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7653 New CDC Event for TTL expired rows [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2209:
URL: https://github.com/apache/phoenix/pull/2209#issuecomment-3046723312
@palashc @haridsv you have any other reviews? I would like to get this in sooner because some more changes are still expected in this area.
For @haridsv's suggestion on the scope, i will create follow-up Jira. This also made me thinking, not having pre-image in scope of the query is sort of not helpful for DELETE and now TTL_DELETE events 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org