You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "askalt (via GitHub)" <gi...@apache.org> on 2026/02/08 09:12:31 UTC
[PR] feat(datafusion): add table provider builder [iceberg-rust]
askalt opened a new pull request, #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123
At the moment, the provider is only available through the Iceberg implementation of the DF catalog, which can be inconvenient when using a custom catalog implementation. This patch adds a public builder, allowing external users to create the provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-4898926135
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): add table provider builder [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-3881523578
Thanks for the example! But I'm still a bit confused about why you need to pass `schema` into `IcebergTableProvider`. The only benefit I'm seeing here is that we can avoid one conversion from iceberg schema to arrow schema. Maybe I'm overlooking something?
Imo, the schema should come from iceberg table metadata otherwise there may be unexpected behaviors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-4377757358
👍 makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-4166586014
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): add table provider builder [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-3874062752
Hi @askalt, thanks for raising this PR!
> when using a custom catalog implementation
I'm assuming you are using a custom `Catalog/SchemaProvider` with `IcebergTableProvider`?
I'm trying to understand your use case better. Some examples on how a custom `Catalog/SchemaProvider` with `IcebergTableProvider` would benefit you would be very helpful!
Also I was wondering if making `IcebergTableProvider::try_new` a `pub` function would be an easier solution for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "askalt (via GitHub)" <gi...@apache.org>.
askalt commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-4191308477
@CTTY Hi! Could you please the PR again please? Simplified as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "derrley (via GitHub)" <gi...@apache.org>.
derrley commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-3978797428
👍 to making try_new pub instead of pub(crate)! Would be a big help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-4975451552
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): add table provider builder [iceberg-rust]
Posted by "askalt (via GitHub)" <gi...@apache.org>.
askalt commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-3877363686
> Hi @askalt, thanks for raising this PR!
>
> > when using a custom catalog implementation
>
> I'm assuming you are using a custom `Catalog/SchemaProvider` with `IcebergTableProvider`?
>
> I'm trying to understand your use case better. Some examples on how a custom `Catalog/SchemaProvider` with `IcebergTableProvider` would benefit you would be very helpful!
>
> Also I was wondering if making `IcebergTableProvider::try_new` a `pub` function would be an easier solution for you?
Hi! We actually use `MemorySchemaProvider` from DF and support iceberg tables creation with some known arrow schema (it is why `IcebergTableProviderBuilder::with_schema` is present). The code looks roughly like:
```rust
async fn create_table(
&self,
ctx: &SessionContext,
name: String,
schema: SchemaRef,
) -> Result<()> {
let inner_schema = arrow_schema_to_schema(&schema)?;
let table_creation = TableCreation::builder()
.name(name.clone())
.schema(inner_schema)
.build();
let table = self
.catalog
.create_table(&self.nmid, table_creation)
.await?;
let table_provider = IcebergTableProviderBuilder::new(
Arc::clone(&self.catalog) as _,
table.identifier().clone(),
)
.with_schema(Some(schema))
.build()
.await?;
session_state.register_table(name, Arc::new(table_provider));
Ok(())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): expor table provider constructor [iceberg-rust]
Posted by "askalt (via GitHub)" <gi...@apache.org>.
askalt commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-3882523955
> Thanks for the example! But I'm still a bit confused about why you need to pass `schema` into `IcebergTableProvider`. The only benefit I'm seeing here is that we can avoid one conversion from iceberg schema to arrow schema. Maybe I'm overlooking something?
>
> Imo, the schema should come from iceberg table metadata otherwise there may be unexpected behaviors.
Yes, it seems we could avoid explicit schema passing. Reworked to just export the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2123: feat(datafusion): export table provider constructor
URL: https://github.com/apache/iceberg-rust/pull/2123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-4627182297
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(datafusion): export table provider constructor [iceberg-rust]
Posted by "toutane (via GitHub)" <gi...@apache.org>.
toutane commented on PR #2123:
URL: https://github.com/apache/iceberg-rust/pull/2123#issuecomment-4369440885
@CTTY Hi! I'm another user who would benefit from this change. We're building a custom catalog integration and currently can't instantiate `IcebergTableProvider` outside the built-in catalog.
This PR looks ready to go - just a simple visibility change. Would be great to see it merged!
Thanks to @askalt for the contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org