[PR] [core] Fix query results being scrambled after querying $audit_log ba… [paimon]

["L-Gryps (via GitHub)" <gi...@apache.org>]

L-Gryps opened a new pull request, #8804:
URL: https://github.com/apache/paimon/pull/8804

    Closes #8803
   
     ### Purpose
   
     When a primary-key table has `'table-read.sequence-number.enabled' = 'true'`, querying its `$audit_log` (or `$binlog`) system table and **then** querying the base table causes:
   
     - Column data to be shifted by one position (the internal `_SEQUENCE_NUMBER` value leaks into the first column, all real columns shift right), or
     - A hard `ClassCastException` when a base column is non-numeric, e.g. `HeapLongVector cannot be cast to BytesColumnVector`, surfaced by Flink as `Recovery is suppressed by NoRestartBackoffTimeStrategy`.
   
     **Root cause**
   
     `AuditLogTable`'s constructor mutates the wrapped `FileStoreTable`'s options map in place when sequence-number reading is enabled:
   
     ```java
     this.wrapped.options().put(CoreOptions.KEY_VALUE_SEQUENCE_NUMBER_ENABLED.key(), "true");
   
     FileStoreTable.options() returns schema().options(), which is the same Map<String, String> reference held by the catalog-cached FileStoreTable. After the first SELECT * FROM t$audit_log, the cached base table permanently
     carries key-value.sequence_number.enabled=true. Subsequent reads of the base table go through KeyValueTableRead.unwrap, which then makes ValueContentRowDataRecordIterator prepend a long _SEQUENCE_NUMBER column to every row.
     The physical row now has one extra column ahead of the logical row type, causing column mis-alignment or vector type mismatch.
   
     Fix
   
     Instead of mutating the shared options map, create a private copy of the wrapped FileStoreTable that carries the internal KEY_VALUE_SEQUENCE_NUMBER_ENABLED=true option only for the audit-log's own reads. The catalog-cached
     base table is left untouched.
   
     wrapped = wrapped.copyWithoutTimeTravel(
             Collections.singletonMap(
                     CoreOptions.KEY_VALUE_SEQUENCE_NUMBER_ENABLED.key(), "true"));
   
     ### Test
   
     Added regression coverage at two layers:
   
     - paimon-core/src/test/java/org/apache/paimon/table/system/AuditLogTableTest.java — testReadBaseTableAfterAuditLogDoesNotIncludeSequenceNumber: constructs an AuditLogTable and asserts the wrapped base table's row type /
       options are not polluted with _SEQUENCE_NUMBER / KEY_VALUE_SEQUENCE_NUMBER_ENABLED.
   
     - paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/BatchFileStoreITCase.java — testReadBaseTableAfterAuditLogWithSequenceNumberEnabled: end-to-end reproduction of the reported user scenario (query
       t$audit_log, then re-query t); base-table result must remain unchanged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Fix query results being scrambled after querying $audit_log ba… [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #8804:
URL: https://github.com/apache/paimon/pull/8804


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Fix query results being scrambled after querying $audit_log ba… [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on PR #8804:
URL: https://github.com/apache/paimon/pull/8804#issuecomment-5057396576

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org