[Polaris Community Sync][Table Source Proposal] October 2, 2025

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi folks,

Thanks to everyone who joined the community meeting about Table Source Proposal.

Here's the main takeaways:
1. I think we have a consensus about the use case about reading
existing Parquet files to easily create Iceberg metadata and so
leverage Polaris features (especially about governance).
2. We don't have yet a clear consensus for "existing Iceberg tables"
(that can be addressed with Catalog Federation) and unstructured data
(PDF files, video, image, ...) needs more discussion.
3. In order to move forward, I propose to focus on the "existing
Parquet files" use case.
4. Then, I'm proposing the following action plan:
4.1. I propose to split the Table Source proposal document, with a
focus on the "Parquet file" use case.
4.2. We discussed leveraging Generic Table and server side scan API
for that. I propose to work with Yun and I will start a PoC to verify
it's a viable option and identify the changes eventually required on
Generic Table.
4.3. Depending of 4.2, I will update the proposal document about
"existing Parquet files" and open a PR with change.

Thoughts ?

Here's the record:
https://drive.google.com/file/d/1x4XjZCop7WaA8L0m81UrepE2nvRsfUU1/view?usp=sharing

I will submit a PR to update the website too and I will update the
corresponding GitHub Issue and proposal document.

Thanks again!

Regards
JB

Re: [Polaris Community Sync][Table Source Proposal] October 2, 2025

[Adam Christian <ad...@gmail.com>]

Thanks, JB! Looking forward to the proposal!

Just for some context on the offline discussion with me, Yun, & Yufei
(since nothing happens until it's on the mailing list :P), we sat down to
discuss some of the active work around the MCP Server and some loose ideas
about how generic tables, volumes, and object tables could overlap. We
didn't reach any consensus about the way forward, but we did recognize that
there was significant overlap in the three concepts.

Per JB's mail, JB & co will be coming up with an updated proposal to
integrate these three concepts (generic tables, volumes, & table sources)
into a consolidated way forward to support some of the overlapping use
cases.

I believe that this next iteration will be very fruitful as we are all
reaching towards the same goal - how do we make the Polaris Catalog API
extend beyond just Iceberg and become a lakehouse catalog for all tables?
I'm excited for the future! This is going to require lots of deep technical
discussion, but this feature set is going to be well worth it!

Go community,

Adam

On Thu, Nov 20, 2025 at 7:03 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi JB,
>
> Thanks so much for the detailed update! The direction looks great, and I
> really appreciate the increased clarity around scope, especially the focus
> on interoperability and the refinement of both the Generic Table and Object
> Table approaches. Highlighting the missing pieces like credential vending
> is also very helpful.
>
> We had a productive discussion with Adam and Yun yesterday that aligns well
> with what you’ve outlined. Concretely, credential vending looks like the
> most important piece to move forward first. For volume support, I had a
> proposal from about a year ago (
>
> https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?usp=sharing
> ) that was discussed across the community, there may be an opportunity to
> consolidate these efforts as the proposal evolves.
>
> Thanks again for driving this!
>
>
> Yufei
>
>
> On Wed, Nov 19, 2025 at 1:40 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi folks,
> >
> > Following the Polaris Table Source Proposal Community Meeting on
> > October 2nd, we conducted extensive investigations based on the
> > valuable feedback provided.
> >
> > We are now working on an updated proposal that consolidates existing
> > features, current proposals, and the feedback received.
> >
> > This update will include:
> > - Clarification of the proposal's purpose and specific use cases.
> > - Identification of which Polaris values this proposal focuses on
> > (interoperability at core).
> > - An analysis of leveraging a Generic Table approach, including pros and
> > cons.
> > - A more detailed section for the Object Table proposal, with more
> > comparison with similar proposals (like the Volume proposal).
> >
> > We also identified some missing components, particularly regarding
> > general credential vending.
> >
> > To move forward, we propose the following next steps:
> > 1. Update the proposal and share the revised version here for review
> > (in the coming days).
> > 2. Start developing a concrete prototype to better illustrate the
> > proposal's concepts.
> > 3. Schedule a new Table Source meeting to discuss the latest version
> > of the proposal in depth.
> >
> > Thank you for your input!
> >
> > Regards,
> > JB
> >
> > On Fri, Oct 3, 2025 at 12:11 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > >
> > > Hi folks,
> > >
> > > Thanks to everyone who joined the community meeting about Table Source
> > Proposal.
> > >
> > > Here's the main takeaways:
> > > 1. I think we have a consensus about the use case about reading
> > > existing Parquet files to easily create Iceberg metadata and so
> > > leverage Polaris features (especially about governance).
> > > 2. We don't have yet a clear consensus for "existing Iceberg tables"
> > > (that can be addressed with Catalog Federation) and unstructured data
> > > (PDF files, video, image, ...) needs more discussion.
> > > 3. In order to move forward, I propose to focus on the "existing
> > > Parquet files" use case.
> > > 4. Then, I'm proposing the following action plan:
> > > 4.1. I propose to split the Table Source proposal document, with a
> > > focus on the "Parquet file" use case.
> > > 4.2. We discussed leveraging Generic Table and server side scan API
> > > for that. I propose to work with Yun and I will start a PoC to verify
> > > it's a viable option and identify the changes eventually required on
> > > Generic Table.
> > > 4.3. Depending of 4.2, I will update the proposal document about
> > > "existing Parquet files" and open a PR with change.
> > >
> > > Thoughts ?
> > >
> > > Here's the record:
> > >
> >
> https://drive.google.com/file/d/1x4XjZCop7WaA8L0m81UrepE2nvRsfUU1/view?usp=sharing
> > >
> > > I will submit a PR to update the website too and I will update the
> > > corresponding GitHub Issue and proposal document.
> > >
> > > Thanks again!
> > >
> > > Regards
> > > JB
> >
>

Re: [Polaris Community Sync][Table Source Proposal] October 2, 2025

[Yufei Gu <fl...@gmail.com>]

Hi JB,

Thanks so much for the detailed update! The direction looks great, and I
really appreciate the increased clarity around scope, especially the focus
on interoperability and the refinement of both the Generic Table and Object
Table approaches. Highlighting the missing pieces like credential vending
is also very helpful.

We had a productive discussion with Adam and Yun yesterday that aligns well
with what you’ve outlined. Concretely, credential vending looks like the
most important piece to move forward first. For volume support, I had a
proposal from about a year ago (
https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?usp=sharing
) that was discussed across the community, there may be an opportunity to
consolidate these efforts as the proposal evolves.

Thanks again for driving this!


Yufei


On Wed, Nov 19, 2025 at 1:40 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi folks,
>
> Following the Polaris Table Source Proposal Community Meeting on
> October 2nd, we conducted extensive investigations based on the
> valuable feedback provided.
>
> We are now working on an updated proposal that consolidates existing
> features, current proposals, and the feedback received.
>
> This update will include:
> - Clarification of the proposal's purpose and specific use cases.
> - Identification of which Polaris values this proposal focuses on
> (interoperability at core).
> - An analysis of leveraging a Generic Table approach, including pros and
> cons.
> - A more detailed section for the Object Table proposal, with more
> comparison with similar proposals (like the Volume proposal).
>
> We also identified some missing components, particularly regarding
> general credential vending.
>
> To move forward, we propose the following next steps:
> 1. Update the proposal and share the revised version here for review
> (in the coming days).
> 2. Start developing a concrete prototype to better illustrate the
> proposal's concepts.
> 3. Schedule a new Table Source meeting to discuss the latest version
> of the proposal in depth.
>
> Thank you for your input!
>
> Regards,
> JB
>
> On Fri, Oct 3, 2025 at 12:11 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> > Hi folks,
> >
> > Thanks to everyone who joined the community meeting about Table Source
> Proposal.
> >
> > Here's the main takeaways:
> > 1. I think we have a consensus about the use case about reading
> > existing Parquet files to easily create Iceberg metadata and so
> > leverage Polaris features (especially about governance).
> > 2. We don't have yet a clear consensus for "existing Iceberg tables"
> > (that can be addressed with Catalog Federation) and unstructured data
> > (PDF files, video, image, ...) needs more discussion.
> > 3. In order to move forward, I propose to focus on the "existing
> > Parquet files" use case.
> > 4. Then, I'm proposing the following action plan:
> > 4.1. I propose to split the Table Source proposal document, with a
> > focus on the "Parquet file" use case.
> > 4.2. We discussed leveraging Generic Table and server side scan API
> > for that. I propose to work with Yun and I will start a PoC to verify
> > it's a viable option and identify the changes eventually required on
> > Generic Table.
> > 4.3. Depending of 4.2, I will update the proposal document about
> > "existing Parquet files" and open a PR with change.
> >
> > Thoughts ?
> >
> > Here's the record:
> >
> https://drive.google.com/file/d/1x4XjZCop7WaA8L0m81UrepE2nvRsfUU1/view?usp=sharing
> >
> > I will submit a PR to update the website too and I will update the
> > corresponding GitHub Issue and proposal document.
> >
> > Thanks again!
> >
> > Regards
> > JB
>

Re: [Polaris Community Sync][Table Source Proposal] October 2, 2025

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi folks,

Following the Polaris Table Source Proposal Community Meeting on
October 2nd, we conducted extensive investigations based on the
valuable feedback provided.

We are now working on an updated proposal that consolidates existing
features, current proposals, and the feedback received.

This update will include:
- Clarification of the proposal's purpose and specific use cases.
- Identification of which Polaris values this proposal focuses on
(interoperability at core).
- An analysis of leveraging a Generic Table approach, including pros and cons.
- A more detailed section for the Object Table proposal, with more
comparison with similar proposals (like the Volume proposal).

We also identified some missing components, particularly regarding
general credential vending.

To move forward, we propose the following next steps:
1. Update the proposal and share the revised version here for review
(in the coming days).
2. Start developing a concrete prototype to better illustrate the
proposal's concepts.
3. Schedule a new Table Source meeting to discuss the latest version
of the proposal in depth.

Thank you for your input!

Regards,
JB

On Fri, Oct 3, 2025 at 12:11 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi folks,
>
> Thanks to everyone who joined the community meeting about Table Source Proposal.
>
> Here's the main takeaways:
> 1. I think we have a consensus about the use case about reading
> existing Parquet files to easily create Iceberg metadata and so
> leverage Polaris features (especially about governance).
> 2. We don't have yet a clear consensus for "existing Iceberg tables"
> (that can be addressed with Catalog Federation) and unstructured data
> (PDF files, video, image, ...) needs more discussion.
> 3. In order to move forward, I propose to focus on the "existing
> Parquet files" use case.
> 4. Then, I'm proposing the following action plan:
> 4.1. I propose to split the Table Source proposal document, with a
> focus on the "Parquet file" use case.
> 4.2. We discussed leveraging Generic Table and server side scan API
> for that. I propose to work with Yun and I will start a PoC to verify
> it's a viable option and identify the changes eventually required on
> Generic Table.
> 4.3. Depending of 4.2, I will update the proposal document about
> "existing Parquet files" and open a PR with change.
>
> Thoughts ?
>
> Here's the record:
> https://drive.google.com/file/d/1x4XjZCop7WaA8L0m81UrepE2nvRsfUU1/view?usp=sharing
>
> I will submit a PR to update the website too and I will update the
> corresponding GitHub Issue and proposal document.
>
> Thanks again!
>
> Regards
> JB