You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "limatakamaidotcom-github (via GitHub)" <gi...@apache.org> on 2026/09/18 13:30:54 UTC

[I] RESTTableScan's PlanTableScanRequest has no snapshotId set by default [iceberg]

limatakamaidotcom-github opened a new issue, #18163:
URL: https://github.com/apache/iceberg/issues/18163

   ### Query engine
   
   Spark
   
   ### Question
   
   Hi Team,
    
   When `spark.sql.catalog.spark_catalog.scan-planning-mode=server`, by default,  RESTTableScan's PlanTableScanRequest has no snapshotId set and is up to the catalog server to pick up the latest snapshotId. In theory, between Spark’s datasource schema binding on client side and remote table scan on the server side, table schema can change and new data can be ingested into the table. The server can return files written with new schema while the client tries to read them with old schema. Is there a potential issue regarding schema mismatch as the result of different snapshotIds are used by the client and the server?
    
    
   The source of the behavior:
    
   Compared with the local mode, DataTableScan.doPlanFiles() invokes the public version of SnapshotScan.snapshot(), which picks snapshotId via table().currentSnapshot() if snapshotId is not provided in the context, so schema binding and table scanning are based on the same snapshotId in the Table object.
    
   public Snapshot snapshot() {
     return snapshotId() != null ? table().snapshot(snapshotId()) : table().currentSnapshot();
   }
    
   In contrast, RESTTableScan.planFiles() invokes the protected version of SnapshotScan.snapshot(), which returns null if not provided in the context.   
    
   protected Long snapshotId() {
     return context().snapshotId();
   }
    
   I wonder if the diverged behavior is intentional? If yes, for what reason?
    
   Thanks,
   Limin    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableScan's PlanTableScanRequest has no snapshotId set by default [iceberg]

Posted by "yusufyildiz34 (via GitHub)" <gi...@apache.org>.
yusufyildiz34 commented on issue #18163:
URL: https://github.com/apache/iceberg/issues/18163#issuecomment-6372027731

   I looked into this and it doesn't look intentional to me. When no snapshot is set, `RESTTableScan.planFiles()` sends no `snapshot-id`, so the server plans against whatever is current on its side, while the client has already resolved the schema and projection from the metadata it loaded. If a commit lands in between, the returned tasks come from a newer snapshot than the one the client resolved. Local planning avoids this because `snapshot()` falls back to `table().currentSnapshot()`, and the spec only describes `snapshot-id` as the snapshot for a point-in-time scan without saying what the server should use when it's absent. A small fix would be to send the client's current snapshot ID (keeping `use-snapshot-schema` false, as today) when neither a snapshot nor an incremental range is set, with a test that commits after creating the scan and checks that the planned files come from the client's snapshot. @singhpk234 does that match the intended behavior for server-side planning? If 
 so, I'd be happy to open a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org