[I] [Bug] Snapshot live manifest entry cache matches on snapshot id only and serves stale entries after a rollback [paimon-cpp]

["wangyong9999 (via GitHub)" <gi...@apache.org>]

wangyong9999 opened a new issue, #299:
URL: https://github.com/apache/paimon-cpp/issues/299

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
   
   ### Paimon-cpp version
   
   main (ded7cfb8)
   
   ### Minimal reproduce step
   
   1. Create a primary key table with one bucket and commit three times (snapshots 1 to 3), for example `(k1, v1), (k2, v2)`, then `(k3, v3)`, then `(k1, v1b)`.
   2. Scan it with a `Cache` set on the `ScanContext` and `scan.manifest-entry-cache.max-snapshots = 2`, so the live manifest entries of snapshot 3 are cached for bucket 0.
   3. Roll the table back and commit twice more, for example from Flink:
   
      ```sql
      CALL sys.rollback_to(`table` => 'default.t', snapshot_id => 1);
      INSERT INTO t VALUES ('k4', 'v4');
      INSERT INTO t VALUES ('k1', 'v1c');
      ```
      The rollback deletes `snapshot-2`, `snapshot-3` and the data files only they referenced; the two inserts create a new `snapshot-2` and `snapshot-3`.
   4. Scan again through the same `Cache`.
   
   ### What doesn't meet your expectations?
   
   The second scan plans from the cached entries of the deleted snapshot 3. Reading `k1` or `k3` fails with `Not exist: File .../bucket-0/data-....parquet not exists`, `k4` returns no row although it was committed, and only `k2` is correct. The table recovers once a fourth snapshot is committed and the cached ids fall behind.
   
   `FileStoreScan::ReadManifestEntriesWithCache` treats `cached->snapshot_id == snapshot.Id()` as a hit (`file_store_scan.cpp:343`). Nothing else identifies the snapshot, so a snapshot id that is reused after a rollback (or after dropping and recreating a table at the same path) matches entries that belong to a different snapshot. With the default `max-snapshots = 0` the cache is off; any deployment that enables it for point lookups is exposed.
   
   ### Anything else?
   
   Storing the snapshot's delta manifest list name next to the cached entries and requiring it to match on a hit is enough: the name carries a UUID, so a rewritten snapshot never matches, and a mismatch falls through to the existing rebuild path.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Snapshot live manifest entry cache matches on snapshot id only and serves stale entries after a rollback [paimon-cpp]

["wangyong9999 (via GitHub)" <gi...@apache.org>]

wangyong9999 closed issue #299: [Bug] Snapshot live manifest entry cache matches on snapshot id only and serves stale entries after a rollback
URL: https://github.com/apache/paimon-cpp/issues/299


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Snapshot live manifest entry cache matches on snapshot id only and serves stale entries after a rollback [paimon-cpp]

["wangyong9999 (via GitHub)" <gi...@apache.org>]

wangyong9999 commented on issue #299:
URL: https://github.com/apache/paimon-cpp/issues/299#issuecomment-5585426986

   Folded into #298 together with the bucket selection problem; one issue and one PR (#302) cover both point-lookup defects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org