You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "shubham-roy (via GitHub)" <gi...@apache.org> on 2026/09/07 10:33:23 UTC
[PR] PHOENIX-8000 Conditional TTL not enforced at compaction for SYSTEM tables [phoenix]
shubham-roy opened a new pull request, #2613:
URL: https://github.com/apache/phoenix/pull/2613
## Problem
`SYSTEM` tables configured with a **conditional TTL** (e.g. `SYSTEM.CDC_STREAM`) never
have that TTL applied at compaction. Expired rows are hidden at read time (masking) but
are **never physically purged** from the HFiles, so they accumulate indefinitely.
## Root cause
In `CompactionScanner.NonPartitionedTableTTLTracker`, `SYSTEM` tables are unconditionally
routed to the HBase column-family **descriptor** TTL instead of the compiled **conditional**
TTL expression:
boolean isSystemTable = pTable.getType() == PTableType.SYSTEM;
if (isSystemTable
|| pTable.getTTLExpression().equals(TTL_EXPRESSION_DEFINED_IN_TABLE_DESCRIPTOR)) {
ttlExpr = TTLExpressionFactory.create(cfd.getTimeToLive()); // FOREVER for a conditional-TTL table
} else {
ttlExpr = ... pTable.getCompiledTTLExpression(pConn) ...
}
A conditional TTL is stored in `SYSTEM.CATALOG`, not the descriptor, so the descriptor TTL
stays `FOREVER`. As a result `ttlExprForRow` is a literal `FOREVER`,
`RowContext.hasConditionalTTL()` is always `false`, and `postProcessForConditionalTTL()`
(the only place expired rows are dropped) never runs. Expiry is enforced only as read-time
masking, never at compaction.
## Fix
Route a `SYSTEM` table through the compiled conditional-TTL path when its TTL is a
`ConditionalTTLExpression`; keep the descriptor path for literal / descriptor-defined TTLs.
boolean isConditionalTTL = pTable.getTTLExpression() instanceof ConditionalTTLExpression;
if ((isSystemTable && !isConditionalTTL)
|| pTable.getTTLExpression().equals(TTL_EXPRESSION_DEFINED_IN_TABLE_DESCRIPTOR)) {
No behavior change for literal/descriptor-defined TTLs or for non-conditional SYSTEM tables.
## Testing
New integration test `CDCStreamCompactionTTLIT` (`phoenix-core/src/it/java/.../end2end/`):
splits a CDC-enabled table to create a **closed** `SYSTEM.CDC_STREAM` partition row
(`PARTITION_END_TIME IS NOT NULL`), advances a `ManualEnvironmentEdge` past the partition
expiry window, major-compacts `SYSTEM.CDC_STREAM`, and asserts the closed-partition row is
**physically removed** (raw HBase row count drops from `total` to `total - closed`).
Written test-first: fails on current code (`expected:<2> but was:<3>`), passes with the fix.
mvn -pl phoenix-core -am -Dit.test=CDCStreamCompactionTTLIT -DfailIfNoTests=false verify
JIRA: https://issues.apache.org/jira/browse/PHOENIX-8000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-8000 Conditional TTL not enforced at compaction for SYSTEM tables [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #2613:
URL: https://github.com/apache/phoenix/pull/2613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-8000 Conditional TTL not enforced at compaction for SYSTEM tables [phoenix]
Posted by "shubham-roy (via GitHub)" <gi...@apache.org>.
shubham-roy commented on PR #2613:
URL: https://github.com/apache/phoenix/pull/2613#issuecomment-5609556356
>not able to see clear diff for your latest commit since it was squashed. We usually keep adding more commits in PR so that it becomes easier for reviewers to know the clear diff.
@virajjasani , I did not notice claude was force pushing. Will update my prompts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-8000 Conditional TTL not enforced at compaction for SYSTEM tables [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2613:
URL: https://github.com/apache/phoenix/pull/2613#issuecomment-5606531402
@shubham-roy not able to see clear diff for your latest commit since it was squashed. We usually keep adding more commits in PR so that it becomes easier for reviewers to know the clear diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-8000 Conditional TTL not enforced at compaction for SYSTEM tables [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2613:
URL: https://github.com/apache/phoenix/pull/2613#issuecomment-5589303625
-1 | checkstyle | 0m 26s | /results-checkstyle-phoenix-core-server.txt | phoenix-core-server: The patch generated 1 new + 32 unchanged - 0 fixed = 33 total (was 32)
-- | -- | -- | -- | --
-1 checkstyle 0m 26s [/results-checkstyle-phoenix-core-server.txt](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-2613/2/artifact/yetus-general-check/output/results-checkstyle-phoenix-core-server.txt) phoenix-core-server: The patch generated 1 new + 32 unchanged - 0 fixed = 33 total (was 32)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-8000 Conditional TTL not enforced at compaction for SYSTEM tables [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2613:
URL: https://github.com/apache/phoenix/pull/2613#issuecomment-5588999288
-1 | spotless | 0m 25s | | patch has 26 errors when running spotless:check, run spotless:apply to fix.
-- | -- | -- | -- | --
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-8000 Conditional TTL not enforced at compaction for SYSTEM tables [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2613:
URL: https://github.com/apache/phoenix/pull/2613#issuecomment-5588993982
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-2613/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org