You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "NoahKusaba (via GitHub)" <gi...@apache.org> on 2026/07/21 02:07:30 UTC
[PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
NoahKusaba opened a new pull request, #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862
## Which issue does this PR close?
https://github.com/apache/iceberg-rust/issues/2861
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## What changes are included in this PR?
<!--
Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
-->
## Are these changes tested?
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5065032513
> Thanks for iterating with me! What are we gaining here over using the StaticTableProvider now?
Thanks for helping me with this too (still fixing the schema update with snapshot id problem), and sorry for responding late.
To answer your question, anyone not using the iceberg-ballista integration likely won't benefit from these changes, and they should keep using IcebergStaticTableProvider for time-travel reads. After this PR the two resolve schemas through the same helper and both refuse writes when pinned, so read semantics are identical.
Ballista requires you instantiate one TableProvider, through which queries are routed (explained above). This creates two requirements:
1. The TableProvider needs to carry all the information required to rebuild it on the scheduler, which de-serializes the logical plan and re-plans it (this PR + another one add the missing requisite information: snapshot-id + catalog-configuration). StaticTableProvider is built from an already-loaded Table and carries no catalog identity, so there is nothing a codec can rebuild it from.
2. If I want the same instantiated TableProvider to allow for both Scanning + Insert + etc, it needs functional .scan() and .insert_into() implementations. StaticTableProvider has the methods (the trait requires them), but its .insert_into() unconditionally errors — it holds no catalog handle to commit through, so writes are structurally impossible for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5040802525
> what the benefit of this is vs using the `StaticTableProvider`?
>
> If we decide this is the right path forward, I wonder if we can make it generic such that this works for incremental append scans etc also? Maybe with a `ScanRange` enum or something? Happy to hear thoughts here!
My idea is to register tables to datafusion using IcebergTableProvider, so that the same registration also allows for inserts/deletes/merges, etc. Whereas StaticTableProvider is strictly scan only.
Also a staticTableProvider can never go back to current_state as it never refreshes, with my PR unpinning
`
with_snapshot_id(None)
`
will return back to current state.
To make Distributed-Datafusion engines work (in our case it's Ballista), a field like Snapshot_id (and in the future PR catalog_type, name, storage_properties) to this IcebergTableProvider, as the parameters need to round-trips through plan serialization to remote executors.
My understanding of how ballista works:
Client -> Register Table -> LogicalCodec (serialize/de-serialze provider data) -> Scheduler -> PhysicalCodec (serialize/de-serialize) -> Executors -> Do the thing
To reiterate IcebergTableProvider will hold the raw recipe (snapshot_id, table_identity, catalogConfig (to be added in a new PR)), to recreate table connections on distributed executors.
It makes slightly more sense in the original datafusion PR I was making (which I'm splitting up)
https://github.com/apache/iceberg-rust/pull/2727
Happy to replace snapshot_id with a ScanRange enum as well, we can even replace it in a future PR without breaking anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5029582237
@CTTY do you have time to look at this?
Tried to make this change as simple as possible. Most lines are just part of the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5063156228
Thanks for iterating with me! What are we gaining here over using the StaticTableProvider now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5057222839
Something seems a little off here to me. Read/write skew seems like a footgun. A single registered table that reads snapshot A but writes to HEAD is a confusing object. `INSERT INTO t; SELECT * FROM t` won't show the row you just inserted. That's a strange contract to hand a user in my opinion.
I also think schema isn't being handled here correctly for pinned snapshots, we should resolve against the snapshot's schema rather than current, the static table provider handles this correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5062798811
> Something seems a little off here to me. Read/write skew seems like a footgun. A single registered table that reads snapshot A but writes to HEAD is a confusing object. `INSERT INTO t; SELECT * FROM t` won't show the row you just inserted. That's a strange contract to hand a user in my opinion.
>
> I also think schema isn't being handled here correctly for pinned snapshots, we should resolve against the snapshot's schema rather than current, the static table provider handles this correctly.
insert-read problem:
Sorry for that, I understand what you are saying on careful review. I was so tunnel visioned on the ballista integration, that I had missed actual users directly calling snapshot_id.
The way I have it wired in my Ballista integration is that snapshot_id is resolved by the scheduler when it encodes the physical plan to executors, so a read -> insert -> read would properly show the new data as the snapshot_id would be reset to head by the scheduler as queries sequentially are made.
I think it makes sense to add a write guard, if the snapshot_id is pinned (not None), to constrain confusing behavior? I added it in a new commit, let me know how you think this should be handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on code in PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#discussion_r3740870777
##########
crates/integrations/datafusion/src/table/mod.rs:
##########
@@ -56,6 +56,31 @@ use crate::physical_plan::scan::IcebergTableScan;
use crate::physical_plan::sort::sort_by_partition;
use crate::physical_plan::write::IcebergWriteExec;
+/// Computes the arrow schema to expose for reading `table` at the given snapshot:
+/// the table's current schema for `None`, that snapshot's schema for `Some`, so
+/// a time-travel read plans against the schema in effect at that snapshot.
+pub fn snapshot_arrow_schema(table: &Table, snapshot_id: Option<i64>) -> Result<ArrowSchemaRef> {
Review Comment:
I found myself needing this snapshot_arrow_schema value also in my physical codec for my iceberg-ballista crate, so I made a generic function to avoid code duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5065904302
Schema Problem:
Following StaticTableProvider's approach (per your suggestion), pinning now re-derives the exposed schema from the pinned snapshot.
While addressing this, I found a second issue: with_snapshot_id cached the Arrow schema at construction time (the current schema). Pinning across a schema evolution would then push down non-existent column names from the old schema.
Fix: Schema resolution is now centralized in a shared snapshot_arrow_schema helper that looks up the snapshot’s own schema via snapshot.schema(metadata). Both with_snapshot_id and IcebergStaticTableProvider constructors use this helper, ensuring a single resolution path. with_snapshot_id is now fallible, as the snapshot ID is validated during pinning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#discussion_r3738655889
##########
crates/integrations/datafusion/src/table/mod.rs:
##########
@@ -56,6 +56,31 @@ use crate::physical_plan::scan::IcebergTableScan;
use crate::physical_plan::sort::sort_by_partition;
use crate::physical_plan::write::IcebergWriteExec;
+/// Computes the arrow schema to expose for reading `table` at the given snapshot:
+/// the table's current schema for `None`, that snapshot's schema for `Some`, so
+/// a time-travel read plans against the schema in effect at that snapshot.
+pub fn snapshot_arrow_schema(table: &Table, snapshot_id: Option<i64>) -> Result<ArrowSchemaRef> {
Review Comment:
This is now public at the crate root (`iceberg_datafusion::snapshot_arrow_schema`, per the `public-api.txt` diff). Is exposing it to external callers intentional, or is it meant to stay an internal helper shared by the two provider constructors? If the latter, `pub(crate)` would keep the public surface matching what's actually meant to be supported.
##########
crates/integrations/datafusion/src/table/mod.rs:
##########
@@ -88,15 +117,37 @@ impl IcebergTableProvider {
// Load table once to get initial schema
let table = catalog.load_table(&table_ident).await?;
- let schema = Arc::new(schema_to_arrow_schema(table.metadata().current_schema())?);
+ let schema = snapshot_arrow_schema(&table, None)?;
Ok(IcebergTableProvider {
catalog,
table_ident,
schema,
+ snapshot_id: None,
})
}
+ /// Pins reads to a snapshot for time travel; `None` (the default) reads the
+ /// current snapshot. A pinned provider is read-only: `insert_into` would
+ /// commit to the current table state, invisible to its own reads, so it
+ /// errors instead.
+ ///
+ /// Reloads metadata from the catalog to validate the snapshot and re-derive
+ /// `schema()` from it. The reload matters because providers handed out by
+ /// the catalog are cached, so a snapshot committed after construction would
+ /// otherwise look nonexistent.
+ pub async fn with_snapshot_id(mut self, snapshot_id: Option<i64>) -> Result<Self> {
+ let table = self.catalog.load_table(&self.table_ident).await?;
+ self.schema = snapshot_arrow_schema(&table, snapshot_id)?;
+ self.snapshot_id = snapshot_id;
+ Ok(self)
+ }
Review Comment:
This reloads table metadata from the catalog purely to validate the snapshot and recompute the schema, and `scan()` (line 178-183 below) reloads again right after. That's two catalog round trips before a query that needs one, and it makes what reads like a builder setter into a fallible, async network call. Per the comment above it, this cost exists specifically to handle providers the catalog hands out and caches across calls. Is that caching concern this type's to own, or should the reload-for-freshness logic live wherever the caching actually happens (the Ballista scheduler side, by the description upthread)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5226595345
Thank you so much for taking the time to look at this both @mbutrovich and @xanderbailey. I understand much better now the conventions of the datafusion eco-system, and how I can address the problem properly.
I'll close the PR and look to pursue the suggestions made of:
- StaticTableProvider + catalog/table identity.
- datafusion-proto codec getting a distinguishing read/write target.
Don't have alot of time this weekend, so I will start putting up the PR's in a few days. Would appreciate any further advice you have to offer regarding shaping the StaticTableProvider PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5093806988
Hey @xanderbailey can you take a look at this again when you have time?
Hopefully it is in a more acceptable state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "corleyma (via GitHub)" <gi...@apache.org>.
corleyma commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5219763618
Being able to timetravel from the catalog is just basic functionality that most other engines / iceberg clients have.
Not a rustacean but I can certify this would be useful even from a non-distributed execution context. We use this functionality heavily in pyiceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5112493757
> This keeps the current public API clear of any runtime errors that may happen as a result of people trying to pin for scans and fail on read
( Did you mean pin for scans and fail on inserts?)
Thanks for the response. Let me give a clearer description of what Ballista is doing
here, since the execution model isn't obvious.
To register a table with a Ballista context, the client provides a `TableProvider`:
```rust
pub async fn register_iceberg_table(
ctx: &SessionContext,
register_name: &str,
config: IcebergCatalogConfig,
namespace: NamespaceIdent,
table: impl Into<String>,
) -> Result<(), DataFusionError> {
let catalog = bridge::build_catalog(&config).await?;
let provider = iceberg_datafusion::IcebergTableProvider::try_new_with_config(
catalog, config, namespace, table,
)
.await
.map_err(bridge::to_df_err)?;
ctx.register_table(register_name, Arc::new(provider))?;
Ok(())
}
```
From there:
```
Client -> LogicalCodec::try_encode (extract serializable fields from the provider)
Scheduler -> LogicalCodec::try_decode (rebuild the provider)
Scheduler -> create_physical_plan()
-> provider.scan() / .insert_into()
-> IcebergTableScan / (IcebergWriteExec -> IcebergCommitExec)
Scheduler -> physical optimizer rules
Scheduler -> split into stages at shuffle boundaries
Scheduler -> per stage: PhysicalCodec::try_encode -> ship to executors
Executor -> decode, run one partition
```
The provider never reaches an executor. It's rebuilt on the scheduler, used for a single
`create_physical_plan()` call, and stays there. Executors only ever receive encoded
`ExecutionPlan` nodes.
**At `try_decode` the codec can't tell a read from a write.** datafusion-proto encodes DML
write targets as synthetic `TableScan` nodes, so scans and inserts arrive in an identical
shape. That rules out having the codec build an `IcebergStaticTableProvider` for reads and
an `IcebergTableProvider` for writes, since the information isn't there.
On the struct as sketched: `storage_factory` is `Arc<dyn StorageFactory>`, a trait object,
so it can't be serialized. And `StorageConfig` is a props map with no scheme field, so
`storage_props` alone doesn't say which backend to rebuild. So "everything apart from
inner is serialisable" unfortunately doesn't hold as written.
That said, I've been investigating your suggestion in this shape:
```rust
pub struct IcebergSerializableTableProvider {
snapshot_id: Option<i64>,
catalog_config: IcebergCatalogConfig, // from the other PR
schema: SchemaRef,
inner: IcebergTableProvider,
}
```
Most methods delegate to `inner`. `scan()` and `schema()` are overridden to apply the
pinned snapshot. This keeps the distributed concerns in the Ballista crate, which I think
is what you're after.
It will still require some of the upstream change listed in NoahKusaba:feature/datafusion-catalog-config-v2 such as:
```rust
// table/mod.rs, to construct `inner`
pub(crate) async fn try_new(..) -> pub
// physical_plan/scan.rs, to rebuild a scan on the executor
pub(crate) fn new(..) -> pub
pub fn with_predicates(..) // new. `new` derives the predicate from
// Expr filters, but on decode we already
// hold a Predicate and can't turn it back
// physical_plan/mod.rs, these are pub(crate) structs and so currently unnameable
pub use commit::IcebergCommitExec;
pub use write::IcebergWriteExec;
pub use metadata_scan::IcebergMetadataScan;
pub use project::PartitionExpr;
```
I still need to more deeply investigate the full lift here.
So the tradeoff is that the wrapper keeps new fields out of `iceberg-datafusion`, at the cost
of an extra abstraction layer + making execution-node constructors API public (which I needed to do anyway).
I'd also note the pieces in these PRs seem useful beyond Ballista.
`IcebergStaticTableProvider` supports time travel but has no catalog backing, so there's
currently no way to pin a snapshot on a catalog-registered table. My original instinct was
to put this in `iceberg-datafusion` so other engines could reuse it and to avoid the extra
abstraction layer, but I understand if you'd rather keep distributed concerns out of the
crate.
On the runtime-error concern specifically: an alternative is to have `insert_into` reset
`snapshot_id` to `None` and reload the schema, rather than erroring. That trades the error
for behavior some users might find surprising, so I don't have a strong preference. Happy
to go whichever way you prefer.
If we don't want to merge the snapshot change, I will persue the thread of keeping more distributed required features in the ballista-iceberg crate instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5094789739
Run with me here for a moment if you will. Does something like this not work for ballista?
```
pub struct IcebergSerializableTableProvider {
table_ident: TableIdent,
metadata: TableMetadata,
storage_props: HashMap<String, String>,
storage_factory: Arc<dyn StorageFactory>,
snapshot_id: Option<i64>,
inner: IcebergStaticTableProvider,
}
```
If we were to add something like this as the table provider, everything here apart from `inner` is serialisable which means you can fully construct it on the "executor side" (sorry not sure what ballista calls these). This keeps the current public API clear of any runtime errors that may happen as a result of people trying to pin for scans and fail on read. My understanding is that the `IcebergStaticTableProvider` is designed for exactly the use-case you're describing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5154992416
@xanderbailey can you look at this again when you have time, to get a resolution on direction?
I still would really like to merge this in, to avoid the suggested extra wrapper tableProvider, just to get snapshot_id attached + forking table-scans.
Sorry for the wall of text.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5213735063
> I'd also note the pieces in these PRs seem useful beyond Ballista.
IcebergStaticTableProvider supports time travel but has no catalog backing, so there's
currently no way to pin a snapshot on a catalog-registered table.
I think this part is maybe achieved with a UDTF or better still an async UDTF when it’s supported in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion) Add snapshot_id parameter to pin table reads [iceberg-rust]
Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba closed pull request #2862: feat(datafusion) Add snapshot_id parameter to pin table reads
URL: https://github.com/apache/iceberg-rust/pull/2862
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org