You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "haridsv (via GitHub)" <gi...@apache.org> on 2026/02/10 05:10:35 UTC
[PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
haridsv opened a new pull request, #2371:
URL: https://github.com/apache/phoenix/pull/2371
## Summary
- Adds new configuration property `phoenix.mutate.preserveOnLimitExceeded` that changes mutation limit behavior, which can be overridden at the connection level via connection properties.
- When enabled, limit checks occur BEFORE state modification, preserving existing buffered mutations
- Introduces `MutationLimitReachedException` (for executeUpdate) and `MutationLimitBatchException` (for executeBatch) to signal recoverable limit conditions
- Allows applications to commit buffered mutations and continue processing instead of losing state
## Test plan
- Added integration tests for row count and byte size limits with preserve mode
- Added tests for executeBatch with autoCommit on/off
- Added tests for edge cases: row merge, conflicting rows (ON DUPLICATE KEY), constructor limit exceeded
- Added test for legacy behavior (preserveOnLimitExceeded=false)
- Verified estimatedSize tracking for conflicting rows
JIRA: [PHOENIX-7759](https://issues.apache.org/jira/browse/PHOENIX-7759)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#issuecomment-3882049918
There us ibe junit test failure:
```
[org.apache.phoenix.end2end.LongViewIndexDisabledBaseRowKeyMatcherIT.testViewsWithExtendedPK](https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-2371/1/testReport/org.apache.phoenix.end2end/LongViewIndexDisabledBaseRowKeyMatcherIT/testViewsWithExtendedPK/)
Error Details
isMultiTenant(false), extendPK(true), partition(640), tenant(3), rowId(6120), pkInfo(ID1,ID2,ID3), testInfo(DECIMAL,DECIMAL,INTEGER), sortInfo(DESC,DESC,DESC)
Stack Trace
java.lang.AssertionError: isMultiTenant(false), extendPK(true), partition(640), tenant(3), rowId(6120), pkInfo(ID1,ID2,ID3), testInfo(DECIMAL,DECIMAL,INTEGER), sortInfo(DESC,DESC,DESC)
at org.junit.Assert.fail(Assert.java:89)
at org.apache.phoenix.end2end.BaseRowKeyMatcherTestIT.createViewHierarchy(BaseRowKeyMatcherTestIT.java:983)
at org.apache.phoenix.end2end.BaseRowKeyMatcherTestIT.createViewHierarchy(BaseRowKeyMatcherTestIT.java:917)
at org.apache.phoenix.end2end.BaseRowKeyMatcherTestIT.testViewsWithExtendedPK(BaseRowKeyMatcherTestIT.java:795)
...
Standard Error
...
2026-02-09T22:47:18,744 ERROR [main] end2end.BaseRowKeyMatcherTestIT(981): 1
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.phoenix.schema.types.PDataType.getDecimalPrecisionAndScale(PDataType.java:767)
at org.apache.phoenix.schema.types.PDecimal.isSizeCompatible(PDecimal.java:335)
at org.apache.phoenix.schema.PTableImpl.newKey(PTableImpl.java:1251)
at org.apache.phoenix.compile.UpsertCompiler.setValues(UpsertCompiler.java:181)
at org.apache.phoenix.compile.UpsertCompiler.access$500(UpsertCompiler.java:125)
at org.apache.phoenix.compile.UpsertCompiler$UpsertValuesMutationPlan.execute(UpsertCompiler.java:1393)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:665)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:612)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:52)
at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:612)
at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:593)
at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:182)
at org.apache.phoenix.end2end.BaseRowKeyMatcherTestIT.upsertTenantViewRows(BaseRowKeyMatcherTestIT.java:428)
at org.apache.phoenix.end2end.BaseRowKeyMatcherTestIT.createViewHierarchy(BaseRowKeyMatcherTestIT.java:972)
at org.apache.phoenix.end2end.BaseRowKeyMatcherTestIT.createViewHierarchy(BaseRowKeyMatcherTestIT.java:917)
at org.apache.phoenix.end2end.BaseRowKeyMatcherTestIT.testViewsWithExtendedPK(BaseRowKeyMatcherTestIT.java:795)
...
```
This can't be related to my change and must be a flapper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#issuecomment-3882042787
Most of checkstyle errors are also in existing lines. Only the below are in the changed lines:
```
./phoenix-core-client/src/main/java/org/apache/phoenix/execute/MutationState.java:2469: colValuesSizeDiff -= (entry.getKey().getEstimatedSize() + oldValue.length);:31: Unnecessary parentheses around assignment right-hand side. [UnnecessaryParentheses]
./phoenix-core-client/src/main/java/org/apache/phoenix/query/QueryServices.java:100: public static final String PRESERVE_MUTATIONS_ON_LIMIT_EXCEEDED_ATTRIB =:17: Redundant 'final' modifier. [RedundantModifier]
```
The second one is following the existing pattern so I will leave it as is for consistency. I will fix the first one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv merged PR #2371:
URL: https://github.com/apache/phoenix/pull/2371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#discussion_r2829866289
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -4870,9 +4866,9 @@ public MutationState addColumn(PTable table, List<ColumnDef> origColumnDefs,
/**
* To check if TTL is defined at any of the child below we are checking it at
* {@link org.apache.phoenix.coprocessor.MetaDataEndpointImpl#mutateColumn(List, ColumnMutator, int, PTable, PTable, boolean)}
- * level where in function
- * {@link org.apache.phoenix.coprocessor.MetaDataEndpointImpl# validateIfMutationAllowedOnParent(PTable, List, PTableType, long, byte[], byte[], byte[], List, int)}
- * we are already traversing through allDescendantViews.
+ * level where in function {@link org.apache.phoenix.coprocessor.MetaDataEndpointImpl#
+ * validateIfMutationAllowedOnParent(PTable, List, PTableType, long, byte[], byte[],
+ * byte[], List, int)} we are already traversing through allDescendantViews.
Review Comment:
@haridsv it seems maybe you are using different JDK to run spotless:apply. Only JDK 8 based spotless fix works fine as of today because it is default JDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#issuecomment-3925811085
> The PR seems to have caused spotless failure on master branch
@virajjasani That is interesting, I went through each of those in the PR validation and all but one was in lines that I didn't touch and strangely, they were not going away even after running spotless apply. Could you help me figure out what it is? I checked the latest master build but not sure how to look for them: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#issuecomment-3925814807
See my above [comment](https://github.com/apache/phoenix/pull/2371#issuecomment-3882026692).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#issuecomment-3882026692
The spotless errors are outside the changed files in the PR and they are coming even though I already ran spotless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#issuecomment-3917130314
The PR seems to have caused spotless failure on master branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "sanjeet006py (via GitHub)" <gi...@apache.org>.
sanjeet006py commented on code in PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#discussion_r2786844384
##########
phoenix-core-client/src/main/java/org/apache/phoenix/execute/MutationState.java:
##########
@@ -2406,36 +2444,56 @@ int[] getStatementIndexes() {
/**
* Join the newRow with the current row if it doesn't conflict with it. A regular upsert
- * conflicts with a conditional upsert
- * @return True if the rows were successfully joined else False
+ * conflicts with a conditional upsert.
+ * @param mutationState if non-null, checks limits before modification and throws
+ * MutationLimitReachedException if size increase would exceed limits
+ * @return the size change (can be 0, positive, or negative) if merged, or null if conflicting
*/
- boolean join(RowMutationState newRow) {
+ Long join(RowMutationState newRow, MutationState mutationState) throws SQLException {
if (isConflicting(newRow)) {
- return false;
+ return null;
}
- // If we already have a row and the new row has an ON DUPLICATE KEY clause
- // ignore the new values (as that's what the server will do).
+
+ // Pre-compute merged results (no side effects - these return new objects)
+ byte[] combinedOnDupKey =
+ PhoenixIndexBuilderHelper.combineOnDupKey(this.onDupKeyBytes, newRow.onDupKeyBytes);
+ int[] mergedIndexes = joinSortedIntArrays(statementIndexes, newRow.getStatementIndexes());
+
+ // Calculate column values size change
+ long colValuesSizeDiff = 0;
if (newRow.onDupKeyBytes == null) {
- // increment the column value size by the new row column value size
- colValuesSize += newRow.colValuesSize;
+ colValuesSizeDiff = newRow.colValuesSize;
for (Map.Entry<PColumn, byte[]> entry : newRow.columnValues.entrySet()) {
- PColumn col = entry.getKey();
- byte[] oldValue = columnValues.put(col, entry.getValue());
+ byte[] oldValue = columnValues.get(entry.getKey());
if (oldValue != null) {
- // decrement column value size by the size of all column values that were replaced
- colValuesSize -= (col.getEstimatedSize() + oldValue.length);
+ colValuesSizeDiff -= (entry.getKey().getEstimatedSize() + oldValue.length);
}
}
}
- // Concatenate ON DUPLICATE KEY bytes to allow multiple
- // increments of the same row in the same commit batch.
- this.onDupKeyBytes =
- PhoenixIndexBuilderHelper.combineOnDupKey(this.onDupKeyBytes, newRow.onDupKeyBytes);
+
+ // Total size change (can be negative)
+ long totalSizeDiff = colValuesSizeDiff
Review Comment:
Seems like this also was an existing gap that the serialized bytes of ON DUPLICATE KEY was not accounted in mutation size along with accounting for merged indexes length.
I wonder if it can turn into backward incompatible change how size of same mutations can increase due to fixing of existing gaps. That means some previous use cases which were succeeding could fail now.
Should we tie up this new behaviour with this change i.e. only account for new size calculations when this feature is enabled?
##########
phoenix-core-client/src/main/java/org/apache/phoenix/execute/MutationState.java:
##########
@@ -600,11 +637,13 @@ private void joinMutationState(TableRef tableRef, MultiRowMutationState srcRows,
if (!conflictingRows.isEmpty()) {
addMutations(dstMutations, tableRef, conflictingRows);
+ // Update estimatedSize only after actual state change
+ estimatedSize += conflictingRowsTotalSize;
Review Comment:
Was this an existing gap which will be fixed with your change as I don't see size of conflicting rows being accounted right now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7759: Preserve buffered mutations when batch size limit is exceeded [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on PR #2371:
URL: https://github.com/apache/phoenix/pull/2371#issuecomment-3877403470
@sanjeet006py Good eye on catching the gaps! Yes, both would be backwards incompatible but I didn't make it conditional because both are pretty corner case scenarios that are very unlikely to be of practical use cases. In fact, it can be considered a good idea if some remote use case is slipping through larger mutations than intended, as they would get caught and will make the developers to make adjustments, after all these can be considered as bug fixes. However, I am open to make them conditional if anyone has a solid counter argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org