Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

> I thought "default-catalog" could be set via the USE command.

Benny, I think this is a misconception or miscommunication. The USE command
has no impact on the `default-catalog` field. In fact, the proposal's
direction is exactly to establish that USE command should influence how
tables are resolved, same like everywhere else. Right now it is not the
case under the current spec.


On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com> wrote:

> > there is no SQL construct today to explicitly set default-catalog
>
> I thought "default-catalog" could be set via the USE command.
>
> I generally agree with Dan about requiring consistent catalog names.  I
> think that's the lesser evil compared to Iceberg specifying how engines
> should resolve identifiers.  Another thing to consider is that identifier
> resolution can be very expensive at query validation time if identifiers
> need to be looked up from a bunch of places.  Hopefully, it should be
> possible to define a view in such a way that identifiers can be resolved on
> the first try.
>
> Benny
>
> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> Hi Rishabh,
>>
>> You're right that the proposal touches on two aspects, and resolution
>> rules are one of them. The other aspect is the proposal's position that
>> table identifiers should be stored in metadata exactly as they appear in
>> the view text (e.g., even if they're two-part or partially qualified),
>> along with their corresponding UUIDs for validation. This applies both to
>> referenced input tables and the storage table identifier in materialized
>> views.
>>
>> We may be able to converge on this storage format even if we haven't yet
>> converged on the resolution fallback rules. I believe both resolution
>> strategies currently being discussed would still lead to storing
>> identifiers in this way.
>>
>> I'm supportive of moving forward with consensus on the identifier storage
>> format. That said, we may continue to run into questions related to
>> resolution during implementation. For example: Should the storage table
>> identifier follow the same default-catalog and default-namespace resolution
>> behavior as other table references?
>>
>> Thanks,
>> Walaa.
>>
>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>> bhatiarishabh63@gmail.com> wrote:
>>
>>> Hello Walaa,
>>>
>>> Thanks for starting this discussion.
>>>
>>> I think we should decouple at least the MV Spec from the proposal to
>>> change the current behavior of view resolution.
>>>
>>> We can continue having the discussion if the current view spec needs to
>>> be changed or not. Based on the decision at a later point if required we
>>> can update the view resolution rule.
>>>
>>>
>>> Thanks,
>>> Rishabh
>>>
>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> Correction of typo: both engines seem to set default-catalog to the
>>>> view catalog if it is defined, or to null if the view catalog is not
>>>> defined.
>>>>
>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> Hi Dan,
>>>>>
>>>>> Thanks again for your response.
>>>>>
>>>>> I agree that catalog renaming is an environmental event, but it's a
>>>>> real one that happens frequently in practice.
>>>>> Saying that the Iceberg spec cannot accommodate something as common as
>>>>> catalog renaming feels very restrictive, and could make the spec less
>>>>> practical, even unusable, for real-world deployments.
>>>>> I’m sharing this from the perspective of a large data lake environment
>>>>> where views are heavily deployed and operationalized.
>>>>>
>>>>> Further, it's worth noting that the table spec is resilient to catalog
>>>>> renaming, but the view spec is not. If we have an opportunity to make the
>>>>> view spec similarly resilient, I wonder why not?
>>>>> Both specifications are deterministic in their definition, but one is
>>>>> more fragile to environmental changes than the other. Improving resilience
>>>>> does not sacrifice determinism. It simply makes views safer and more
>>>>> portable over time.
>>>>>
>>>>> Separately, given that there is no SQL construct today to explicitly
>>>>> set default-catalog at creation time, what is the intuition behind how
>>>>> engines like Spark and Trino currently assign default-catalog?
>>>>> Today, both engines seem to set default-catalog to null if the view
>>>>> catalog is defined, or to the view catalog if not.
>>>>> What was the intended thought process behind this behavior?
>>>>>
>>>>> Thanks,
>>>>> Walaa
>>>>>
>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dw...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Walaa,
>>>>>>
>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>
>>>>>> This problem stems from the exact environmental/configuration issue
>>>>>> that we should not be trying to address.  I don't think we would expect
>>>>>> references to survive a catalog rename.  That's not something covered by
>>>>>> the spec and needs to be handled separately as a platform-level migration
>>>>>> specific to the affected environment.
>>>>>>
>>>>>> The identifier resolution logic is clear and deterministic.  It
>>>>>> should not matter whether an engine resolves and encodes the
>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>
>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>> something we should be trying to define here.
>>>>>>
>>>>>> -Dan
>>>>>>
>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Dan,
>>>>>>>
>>>>>>> Thanks for chiming in.
>>>>>>>
>>>>>>> I believe the issues we’re seeing now go beyond just catalog naming
>>>>>>> consistency. The behavior around default-catalog itself introduces
>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>> For example:
>>>>>>>
>>>>>>> * When default-catalog is set to null, tables inside views remain
>>>>>>> reachable after a catalog rename. But if it is set to a non-null value,
>>>>>>> table references will break.
>>>>>>>
>>>>>>> * default-catalog causes table references inside views to be early
>>>>>>> bound (i.e., bound at view creation time, especially when using a non-null
>>>>>>> value), while table references inside standalone queries are late bound
>>>>>>> (bound at query time). This creates inconsistencies when resolving the same
>>>>>>> table name inside and outside views, even within the same job.
>>>>>>>
>>>>>>> * It causes Spark's and Trino behavior to drift from the spec. There
>>>>>>> is no way to fully align Spark's behavior without making invasive changes
>>>>>>> to the Spark SQL grammar and the View DataSource API (specifically on the
>>>>>>> CREATE side). This challenge would extend to other engines too. Both Spark
>>>>>>> and Trino set this field based on a heuristic in today's implementation.
>>>>>>>
>>>>>>> * With view nesting (views depending on views), these
>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>
>>>>>>> * It will be difficult to migrate Hive views to Iceberg with that
>>>>>>> model. Migrated Hive views will have to unfollow that spec.
>>>>>>>
>>>>>>> How would you suggest approaching the engine-level changes required
>>>>>>> to support the current default-catalog field?
>>>>>>> Also, do you believe the Spark and Trino communities would align
>>>>>>> around having table resolution behave inconsistently between queries and
>>>>>>> views, or inconsistency between Iceberg and other types of views?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Walaa
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <dw...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was
>>>>>>>> introduced, but I think we need to step back and align on what we are
>>>>>>>> really attempting to support in the spec.
>>>>>>>>
>>>>>>>> The issues we're discussing largely stem from using multiple
>>>>>>>> engines with cross catalog references and configurations where catalog
>>>>>>>> names are not aligned.  If we have multiple engines that all have the same
>>>>>>>> catalog names/configurations, the current spec implementation is well
>>>>>>>> defined for table resolution even across catalogs.  The 'default-catalog'
>>>>>>>> (and namespace equivalent) was intended to address the resolution within
>>>>>>>> the context of the sql text, not to address catalog/naming inconsistencies.
>>>>>>>>
>>>>>>>> I feel like we're trying to adapt the original intent to address
>>>>>>>> the catalog naming/configuration and would argue that we shouldn't attempt
>>>>>>>> to do that as part of the spec.  Inconsistently named catalogs are a
>>>>>>>> reality, but we should consider that a configuration/environmental issue,
>>>>>>>> not something to solve for in the spec.
>>>>>>>>
>>>>>>>> We should support and advocate for consistency in catalog naming
>>>>>>>> and define the spec along those lines.  The fact is that with all of the
>>>>>>>> recent work that's gone into making catalogs pluggable, it makes more sense
>>>>>>>> to just register catalog configuration with consistent names (even if you
>>>>>>>> have to duplicate the configuration for supporting existing
>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>
>>>>>>>> If the goal is to create clever ways to hack the late binding
>>>>>>>> resolution to swap in different catalogs or make references contextual, I
>>>>>>>> feel like that is something we should strongly discourage as it leads to
>>>>>>>> confusion about what is resolved as part of the query.
>>>>>>>>
>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>
>>>>>>>> -Dan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>
>>>>>>>>> I think the intention with the "default-catalog" was that every
>>>>>>>>> query engine uses it to store its session default catalog at the time of
>>>>>>>>> creating the view. This way the view could be reused in another session.
>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>> default-catalog.
>>>>>>>>>
>>>>>>>>> Generally we have different environments we want to support with
>>>>>>>>> the view spec:
>>>>>>>>>
>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>
>>>>>>>>> When the environment supports it, using consistent catalog names
>>>>>>>>> can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>> any issues.
>>>>>>>>>
>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>
>>>>>>>>> This can be the case when different query engines refer to the
>>>>>>>>> same physical catalog by different names. This often happens because
>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What do you think of introducing a view property that specifies if
>>>>>>>>> the "default-catalog" or the "view catalog" should be used? This way, you
>>>>>>>>> could use the "default-catalog" in environments where you can guarantee
>>>>>>>>> consistent naming, but you would be able to directly fallback to the
>>>>>>>>> "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Jan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>
>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>
>>>>>>>>> * Remove default-catalog and default-namespace fields from the
>>>>>>>>> view spec completely.
>>>>>>>>>
>>>>>>>>> * Hence, we do not attempt to define separate view-level default
>>>>>>>>> catalogs or namespaces.
>>>>>>>>>
>>>>>>>>> Instead:
>>>>>>>>>
>>>>>>>>> * If a table identifier inside a view lacks a catalog qualifier,
>>>>>>>>> engines should resolve it using the current engine catalog at query time.
>>>>>>>>>
>>>>>>>>> * Reference table identifiers in the metadata exactly as they
>>>>>>>>> appear in the view SQL text.
>>>>>>>>>
>>>>>>>>> * If an identifier lacks the catalog part at creation, it should
>>>>>>>>> still lack a catalog in the stored metadata.
>>>>>>>>>
>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Walaa.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the
>>>>>>>>>> fallback creates. Also just to be clear, at this point and after clarifying
>>>>>>>>>> the current spec intentions, I am convinced that we should remove the
>>>>>>>>>> default catalog and default namespace fields altogether.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Walaa.
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>
>>>>>>>>>>> - I don't particularly like the current behavior of "default to
>>>>>>>>>>> the view's catalog when default-catalog is not set".  Fundamentally, I
>>>>>>>>>>> believe the intent of default-catalog and default-namespace is there to
>>>>>>>>>>> help users write more concise SQL.
>>>>>>>>>>> - spark session catalog is engine specific and I don't think we
>>>>>>>>>>> should design something that says first use this catalog, then that
>>>>>>>>>>> catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no value
>>>>>>>>>>> in putting views in the catalog to share with other engines
>>>>>>>>>>> - Interoperability between different engine types is very hard
>>>>>>>>>>> due to dialect issues... so I think we should focus on supporting different
>>>>>>>>>>> clusters of the same engine type on a shared catalog.  For example, AI and
>>>>>>>>>>> BI clusters on Spark sharing the same views in a REST catalog.
>>>>>>>>>>>
>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the lines
>>>>>>>>>>> of something Russell proposed last year:
>>>>>>>>>>>
>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>
>>>>>>>>>>> We've been looking at this interoperable identifier problem
>>>>>>>>>>> through the lens of catalog resolution but maybe the right approach is
>>>>>>>>>>> really about templating.
>>>>>>>>>>>
>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a view to
>>>>>>>>>>> span catalogs to support non-Iceberg tables.   Also, the default-catalog
>>>>>>>>>>> property could be templated as well.
>>>>>>>>>>>
>>>>>>>>>>> Thoughts?
>>>>>>>>>>> Benny
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark implementation
>>>>>>>>>>>> conform to the spec? Do we need Spark SQL extensions and/or Spark catalog
>>>>>>>>>>>> APIs for that?
>>>>>>>>>>>>
>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I shared
>>>>>>>>>>>> regarding many resolution methods not consistently being followed in
>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <st...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>> not defined. Current view spec says the fallback should be the
>>>>>>>>>>>>> catalog
>>>>>>>>>>>>> where the view is defined. It doesn't really matter what the
>>>>>>>>>>>>> catalog
>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>> - If a view refers to the tables in the same catalog, this is a
>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>> - If a view refers to tables from another catalog, catalog
>>>>>>>>>>>>> names
>>>>>>>>>>>>> should be included in the reference name already. So no
>>>>>>>>>>>>> ambiguity
>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only recommend
>>>>>>>>>>>>> that
>>>>>>>>>>>>> catalog should be named consistently across usage for better
>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This proposal is to change the fallback behavior to engine's
>>>>>>>>>>>>> session
>>>>>>>>>>>>> default catalog. I am not sure it is better than the current
>>>>>>>>>>>>> fallback
>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>
>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this idea.
>>>>>>>>>>>>> Spark resolves table identifiers during view creation using the session’s
>>>>>>>>>>>>> default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would argue that is a Spark implementation issue for not
>>>>>>>>>>>>> conforming
>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to
>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view
>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist before its
>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible in Spark’s
>>>>>>>>>>>>> implementation, where `default-catalog` is not used.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting would
>>>>>>>>>>>>> require extending SQL syntax and engine APIs to promote it to a first-class
>>>>>>>>>>>>> view concept. This would be intrusive, non-intuitive, and realistically
>>>>>>>>>>>>> very difficult to standardize across engines.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this idea.
>>>>>>>>>>>>> Spark resolves table identifiers during view creation using the session’s
>>>>>>>>>>>>> default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time
>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding model between
>>>>>>>>>>>>> tables vs views (early vs late), and between tables in views and in queries
>>>>>>>>>>>>> (again early vs late). For example, views and tables in queries can
>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when they are used
>>>>>>>>>>>>> inside views -- it even applies to views inside views, which makes this
>>>>>>>>>>>>> very hard to reason about considering nesting.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW statement the
>>>>>>>>>>>>> query engine knowns which catalog the view is being created in. So even
>>>>>>>>>>>>> though we typically use late binding to resolve the view catalog at query
>>>>>>>>>>>>> time, it can also be used at creation time.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> The query engine would need to keep track of the "view
>>>>>>>>>>>>> catalog" where the view is going to be created in. It can use that catalog
>>>>>>>>>>>>> to resolve partial table identifiers if "default-catalog" is not set.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where partial
>>>>>>>>>>>>> identifiers in the view query resolve to a different catalog compared to
>>>>>>>>>>>>> using them outside of a view.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT * from
>>>>>>>>>>>>> sales.orders;
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>> "catalogA", which is the view's catalog.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> For example, if we want to validate that the tables
>>>>>>>>>>>>> referenced in the view exist, how can we do that when default-catalog isn't
>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet?
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we
>>>>>>>>>>>>> validate that a table exists without a default catalog, or do we always use
>>>>>>>>>>>>> the current session catalog?
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> I think we still share the same understanding. Just to
>>>>>>>>>>>>> clarify: when I referred to late binding as “similar” to the proposal, I
>>>>>>>>>>>>> was acknowledging the distinction between view-level and table-level
>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> That said, this still raises an interesting question and a
>>>>>>>>>>>>> potential gap: if default-catalog is only defined at query time, how should
>>>>>>>>>>>>> resolution work during view creation? For example, if we want to validate
>>>>>>>>>>>>> that the tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>> yet?
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view catalog is
>>>>>>>>>>>>> resolved at query time.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish between
>>>>>>>>>>>>> the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>> still work.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late binding
>>>>>>>>>>>>> applies to the view catalog name and by extension to all partial table
>>>>>>>>>>>>> references when no "default-catalog" is present. Resolving the view catalog
>>>>>>>>>>>>> name at query time is not opposed to storing the view metadata in a catalog.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced in a
>>>>>>>>>>>>> SQL statement. That statement contains the information for the query engine
>>>>>>>>>>>>> to resolve the catalog of the view.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT * FROM
>>>>>>>>>>>>> catalogB.namespace.view (after renaming catalogA to catalogB, and keeping
>>>>>>>>>>>>> everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to me
>>>>>>>>>>>>> like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of non-determinism.
>>>>>>>>>>>>> Let me try to address your questions.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec: The
>>>>>>>>>>>>> view is executed when it's being referenced in a SQL statement. That
>>>>>>>>>>>>> statement contains the information for the query engine to resolve the
>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> I think your point is really valuable because the
>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For example
>>>>>>>>>>>>> for the following statement:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>> "catalogA".
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> However, I like the current design of the view spec,
>>>>>>>>>>>>> because it has the "closure" property. Because of the fact that the "view
>>>>>>>>>>>>> catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point before
>>>>>>>>>>>>> going deeper:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback
>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table identifiers alone,
>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed fallback to the
>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help clarify
>>>>>>>>>>>>> two things in the context of the current spec:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog” stored, so
>>>>>>>>>>>>> that an engine knows to fall back to it if default-catalog is null?
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the session's
>>>>>>>>>>>>> default catalog (i.e., without specifying a catalog) in the current Iceberg
>>>>>>>>>>>>> spec?
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> These questions are important because if we can’t
>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then defaulting to
>>>>>>>>>>>>> it is problematic. And if views can't be created in the default catalog,
>>>>>>>>>>>>> then the fallback rule doesn’t generalize.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood correctly,
>>>>>>>>>>>>> you proposal is composed of three parts:
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the current
>>>>>>>>>>>>> session setup is in my opinion the definition of "non-determinism". You
>>>>>>>>>>>>> could be running the same query-engine and catalog-setup on different days,
>>>>>>>>>>>>> with different default session catalogs (which is rather common), and would
>>>>>>>>>>>>> be getting different results.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view always
>>>>>>>>>>>>> produces the same results. The current behavior has some rough edges in
>>>>>>>>>>>>> very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup more
>>>>>>>>>>>>> robust. Which is great. However, having UUID validation still requires us
>>>>>>>>>>>>> to have a portable table identifier specification. Even if we have the
>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply isn't an
>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog interface is defined
>>>>>>>>>>>>> in terms of table identifiers.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and
>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for
>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving
>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text with table
>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use the catalog
>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret or use.
>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated catalog that
>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In such
>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in `my_catalog.namespace1.table1`)
>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to underlying
>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the catalog
>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs actually behave
>>>>>>>>>>>>> in practice.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have some
>>>>>>>>>>>>> concerns with resolving catalog names on the fly with query engine defined
>>>>>>>>>>>>> catalog names. This introduces some flexibility at first glance, but also
>>>>>>>>>>>>> makes misconfiguration difficult to explain.
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and
>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a vote next
>>>>>>>>>>>>> week.
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The proposal
>>>>>>>>>>>>> outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>> comments.
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog names
>>>>>>>>>>>>> in the spec refer to logical catalogs—i.e., the first part of a three-part
>>>>>>>>>>>>> identifier. These correspond to Spark's DataSourceV2 catalogs, Trino
>>>>>>>>>>>>> connectors, and similar constructs. This is a level of abstraction above
>>>>>>>>>>>>> physical catalogs, which are not referenced or used in the view spec. The
>>>>>>>>>>>>> reason is that table identifiers in the view definition/text itself refer
>>>>>>>>>>>>> to logical catalogs, not physical ones (since they interface directly with
>>>>>>>>>>>>> the engine and not a specific metastore).
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think view
>>>>>>>>>>>>> portability is a very important topic for us to continue discussing as it
>>>>>>>>>>>>> relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may
>>>>>>>>>>>>> impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard Tudenhöfner <
>>>>>>>>>>>>> etudenhoefner@apache.org> wrote:
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've added
>>>>>>>>>>>>> a few comments to get a better understanding of how this will look like in
>>>>>>>>>>>>> the actual implementation.
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin
>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our discussion on
>>>>>>>>>>>>> how to reference table identifiers from Iceberg metadata, a key aspect of
>>>>>>>>>>>>> the view specification, particularly in relation to the MV (materialized
>>>>>>>>>>>>> view) extensions.
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a few
>>>>>>>>>>>>> community members to better understand how the current spec is being
>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look
>>>>>>>>>>>>> forward to your feedback and working together to move this forward so we
>>>>>>>>>>>>> can finalize the MV spec as well.
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>
>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

Hi Everyone,

I have discussed this with some community members including Ryan, Dan,
Amogh and Steven. We agreed that we can separate the concerns of table
resolution semantics from the identifier representation in the metadata.
What we need to unblock the MV spec is to decide on the identifier
representation aspect, and not necessarily the entire semantics around
resolution.

For the identifier representation, I think as long as the format of an
identifier in the metadata matches the respective format of the same
identifier in the view definition (e.g., whether it is 1, 2, or 3 part
identifier), we should be fine. This should apply to both view children and
storage table identifiers.

Thanks,
Walaa.

On Mon, May 19, 2025 at 10:54 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> Hi all,
>
> I want to clarify the scope and intent of the proposal, as well as my
> position in light of the recent discussion.
>
> The proposal specifically focuses on how to represent table identifiers
> when they appear in* Iceberg view metadata, not in table metadata*.
>
> This includes cases such as:
> (1) the storage table identifier for materialized views, and
> (2) child tables or views when stored on the view side for lineage or
> dependency tracking.
>
> It does not necessarily apply to view children stored in table metadata
> for freshness tracking. That’s a separate layer of the system, though some
> of the same principles might apply.
>
> The core motivation here is to promote consistency in table resolution
> behavior. Introducing different resolution rules for different types of
> identifiers even within the view itself (e.g., storage table vs. child
> views) would only compound this inconsistency. That’s something I’d
> strongly prefer to avoid.
>
> Finally, this discussion raises an important point: even if we agree on a
> common format for storing table identifiers, we still need to define how
> they are resolved, particularly in the case of the storage table identifier.
> If we document that resolution must go through default-catalog and
> default-namespace, we risk institutionalizing a resolution rule that has
> not yet reached consensus, especially if that behavior needs to be captured
> in new documentation or specs. So while aligning on format is helpful, I’m
> not yet sure it fully unblocks the spec without clearer agreement on the
> associated resolution semantics.
>
> Thanks,
> Walaa.
>
> On Tue, May 13, 2025 at 9:39 AM Steven Wu <st...@gmail.com> wrote:
>
>> > 1. Are we talking about table identifiers in general or are we talking
>> only about the storage table identifier or the source table identifiers?
>>
>> At least I thought Walaa's latest question is about the source table
>> identifiers for the refresh-state metadata in storage table.
>>
>> > 2. When using optional fields for namespace and catalog, what are we
>> using when these fields are not set? (Early vs late binding)
>>
>> I believe we are not talking about the table identifier resolution (early
>> vs late) for the regular view spec anymore, which was the original
>> intention of this thread.
>>
>> On Mon, May 12, 2025 at 11:56 PM Jan Kaul <ja...@mailbox.org.invalid>
>> wrote:
>>
>>> I think it makes sense to distinguish between the identifiers for the
>>> storage table and the source tables.
>>>
>>> *Storage table*: The table that stores the precomputed data of the
>>> materialized view. The storage table identifier is part of the view
>>> metadata and is used to access the precomputed data from the view.
>>>
>>> *Source table*(Base table): Is a table referenced in the materialized
>>> view query. Its state is stored in the "refresh state"(lineage) as part of
>>> the storage table metadata to determine the freshness of the materialized
>>> view.
>>>
>>> And we need to clarify if the current discussion is about table
>>> identifiers in general or if we're discussing either the storage table or
>>> source table identifiers. The following is my opinion on these:
>>>
>>> 1. Storage table identifier
>>>
>>> It is functionally required to store a table identifier as part of the
>>> view metadata to be able to fetch the storage table metadata from the
>>> catalog. The catalog interface uses identifiers, so there is no alternative
>>> here. The question remains if the identifier should use optional namespace
>>> and catalog fields and what to do if these fields are not set. (I'm not
>>> sure if Walaa was also proposing the add an UUID field)
>>>
>>> 2. Source table identifiers
>>>
>>> The source table identifiers can be obtained from parsing the SQL
>>> definition of the view, so it's not strictly required to store them as part
>>> of the metadata. At some point we agreed to use SQL parsing to obtain these
>>> identifiers, because this fits to how the query engine is executing the
>>> query. Moreover, this lets us use different representations for different
>>> query engines. If the representation translation tool supports some kind of
>>> mapping, we could have different "catalog aliases" with different engines.
>>> We agreed to use UUIDs to reference source tables in the refresh-state.
>>> The discussion steven mentioned, was about adding the source table
>>> identifiers to the refresh-state(lineage) for readability, but not to be
>>> used by engines.
>>> The current design doesn't allow multiple representations for the
>>> refresh-state(lineage). This makes it more difficult to solve the "catalog
>>> alias" problem as part of the refresh-state. To avoid this problem and
>>> since the table identifiers are not strictly required, I was advocating to
>>> not store any source table identifiers in the refresh-state. In my opinion,
>>> having one representation of source table identifiers in the refresh-state
>>> is in conflict with having multiple representations for the SQL query,
>>> where the identifiers occur.
>>>
>>> My current questions are:
>>>
>>> 1. Are we talking about table identifiers in general or are we talking
>>> only about the storage table identifier or the source table identifiers?
>>>
>>> 2. When using optional fields for namespace and catalog, what are we
>>> using when these fields are not set? (Early vs late binding)
>>>
>>> Thanks,
>>>
>>> Jan
>>> On 5/12/25 18:23, Steven Wu wrote:
>>>
>>> Regarding the following representation for the storage table identifier:
>>>
>>> * catalog name: optional
>>> * namespace: optional
>>> * table name: required
>>> * UUID: required
>>> It is the same discussion as this github thread.
>>> https://github.com/apache/iceberg/pull/11041/files#r1869998317
>>>
>>> I was in favor of adding them mostly from human readability when
>>> manually inspecting the storage table to avoid a reverse lookup from
>>> table UUID to table identifier.  Benny (and a few others) have expressed
>>> that it may add more ambiguity (like this thread was discussing). To
>>> clarify, we only need to include the identifier as they are defined in the
>>> SQL query (e.g. catalog and namespace are optional).  I doubt it would add
>>> more ambiguity. But they are certainly not functionally required. So I am
>>> not opinionated on this one.
>>>
>>> On Mon, May 12, 2025 at 5:01 AM Jan Kaul <ja...@mailbox.org.invalid>
>>> <ja...@mailbox.org.invalid> wrote:
>>>
>>>> I have a question regarding the following representation for the
>>>> storage table identifier:
>>>>
>>>> * catalog name: optional
>>>> * namespace: optional
>>>> * table name: required
>>>> * UUID: required
>>>>
>>>> Having optional fields as part of the identifier again raises the
>>>> question of what to use if these optional fields are not set. And we again
>>>> have the early vs late binding discussion. Where early binding would
>>>> default to the view catalog and late binding would use the current session
>>>> catalog/namespace.
>>>>
>>>> @Walaa: Which behavior did you have in mind?
>>>>
>>>> In my opinion, having the functionality of MVs depend on the current
>>>> session context is not desirable. So I would argue against late binding.
>>>>
>>>> Best wishes,
>>>>
>>>> Jan
>>>> On 5/8/25 19:59, Walaa Eldin Moustafa wrote:
>>>>
>>>> Let us continue here for the context and continuity of the conversation.
>>>>
>>>> On Thu, May 8, 2025 at 10:55 AM Steven Wu <st...@gmail.com> wrote:
>>>>
>>>>> For the storage table identifier discussion, can we move to Jan's PR
>>>>> on materialized view spec?
>>>>> https://github.com/apache/iceberg/pull/11041
>>>>>
>>>>>
>>>>> On Thu, May 8, 2025 at 10:40 AM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> It looks like we may not reach alignment on the resolution method in
>>>>>> the short term. That said, perhaps we can move forward by aligning on the
>>>>>> format used to store table identifiers in metadata instead.
>>>>>>
>>>>>> The idea is that the same format could work under either resolution
>>>>>> model, which would allow us to make progress independently of that debate.
>>>>>>
>>>>>> Is this storage format of the storage table identifier agreeable to
>>>>>> you?
>>>>>> * catalog name: optional
>>>>>> * namespace: optional
>>>>>> * table name: required
>>>>>> * UUID: required
>>>>>> With the condition that the components included in a table identifier
>>>>>> in the metadata should match those used in the view text for that table.
>>>>>>
>>>>>> The same structure above could apply to storing child tables or view
>>>>>> identifiers as part of lineage, if and when that’s needed.
>>>>>>
>>>>>> For the longer term resolution model, it might make sense to initiate
>>>>>> a cross-community discussion.
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa.
>>>>>>
>>>>>> On Thu, May 8, 2025 at 9:07 AM Steven Wu <st...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> If a table referenced by the view SQL query is deleted and
>>>>>>> recreated, is the regular view still valid? If yes, it won't be correct to
>>>>>>> use resolved UUID to validate the name reference. On the materialized view
>>>>>>> side, we are aligned that UUID is necessary to detect this scenario and
>>>>>>> mark the refresh state "stale" if a referenced table is re-created.
>>>>>>>
>>>>>>> > esolution using default-catalog and default-namespace does not
>>>>>>> provide full determinism,
>>>>>>>
>>>>>>> With the new proposal, the catalog and namespace depends on the
>>>>>>> engine's current context. if the user changes the current context (catalog
>>>>>>> and database) via SET/USE , the view SQL query may resolve to a different
>>>>>>> table. That is probably a bigger problem.
>>>>>>>
>>>>>>> With the current view spec, the intention is set in the view
>>>>>>> creation time. I would rather have the view producer set the explicit
>>>>>>> intention (than the implicit dependency on the engine context at view
>>>>>>> consumption phase).
>>>>>>>
>>>>>>> Also the view SQL can still reference a table with a catalog name
>>>>>>> component.  Hence the catalog naming still matters.The new proposal won't
>>>>>>> really solve the naming problem.
>>>>>>>
>>>>>>> On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Dan,
>>>>>>>>
>>>>>>>> Thanks for the clarification! I agree that UUIDs should not be
>>>>>>>> stored in the SQL definition of the view.
>>>>>>>>
>>>>>>>> Just to clarify: the proposal isn't about modifying the view
>>>>>>>> definition itself, but rather about how table identifiers are represented
>>>>>>>> in Iceberg view metadata, which naturally depends on how those identifiers
>>>>>>>> are resolved.
>>>>>>>>
>>>>>>>> For example, in order to store the "storage table identifier" in
>>>>>>>> the metadata:
>>>>>>>> * Should we apply the same resolution rules based on
>>>>>>>> default-catalog and default-namespace?
>>>>>>>> * And if that resolution method is considered deterministic enough,
>>>>>>>> would you be comfortable not storing the UUID in the storage table
>>>>>>>> identifier metadata?
>>>>>>>>
>>>>>>>> The same questions apply when storing child views or tables as part
>>>>>>>> of the view’s lineage metadata. It would be helpful to clarify the
>>>>>>>> resolution model and whether UUIDs should be part of that for validation or
>>>>>>>> drift detection.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Walaa.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I don't think we want to include the resolved table UUIDs in the
>>>>>>>>> view definition, but rather in the storage table state.  You can still
>>>>>>>>> resolve whether those drift at some point, but I don't feel like it's a
>>>>>>>>> good idea to capture data in the view that we may allow to drift if there
>>>>>>>>> isn't any requirement that they match.  This also aligns with the
>>>>>>>>> identifier resolution being late binding.
>>>>>>>>>
>>>>>>>>> -Dan
>>>>>>>>>
>>>>>>>>> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Steven! So would you agree that resolution using
>>>>>>>>>> default-catalog and default-namespace does not provide full determinism,
>>>>>>>>>> and requires a supporting safety mechanism?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Walaa.
>>>>>>>>>>
>>>>>>>>>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> > If the current model is considered deterministic, do you think
>>>>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>>>>>
>>>>>>>>>>> I am fine with storing UUIDs for table identifiers in the view.
>>>>>>>>>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>>>>>>>>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>>>>>>>>>> fail the query if mismatch.
>>>>>>>>>>>
>>>>>>>>>>> The UUID change doesn't really change the table identifier
>>>>>>>>>>> resolution rule though. It is more of a safety protection.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Steven,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>
>>>>>>>>>>>> > I agree with Dan that we shouldn't solve catalog naming in
>>>>>>>>>>>> the Iceberg view spec.
>>>>>>>>>>>>
>>>>>>>>>>>> To clarify, I don't believe the proposal is trying to solve
>>>>>>>>>>>> catalog naming. What it’s doing is simply this:
>>>>>>>>>>>>
>>>>>>>>>>>> * Proposing that table names inside views resolve the same way
>>>>>>>>>>>> as they do elsewhere (e.g., queries).
>>>>>>>>>>>> * Adopting a model that is already widely used and supported in
>>>>>>>>>>>> the existing ecosystem, which allows for:
>>>>>>>>>>>>     -- Renaming catalog aliases
>>>>>>>>>>>>     -- Swapping catalog implementations behind consistent names
>>>>>>>>>>>>     -- Having different default catalog names across engines
>>>>>>>>>>>> that still point to the same underlying tables
>>>>>>>>>>>>
>>>>>>>>>>>> These are common patterns in production data lakes. Saying
>>>>>>>>>>>> Iceberg views cannot operate in those environments feels unrealistic. In
>>>>>>>>>>>> practice, it means the spec breaks down in situations that users encounter
>>>>>>>>>>>> regularly.
>>>>>>>>>>>>
>>>>>>>>>>>> > The recommendation of using engines’ current catalog and
>>>>>>>>>>>> database can cause context-dependent resolution results.
>>>>>>>>>>>>
>>>>>>>>>>>> * As noted in the doc and earlier replies, fixing a catalog
>>>>>>>>>>>> name doesn’t actually guarantee determinism either. All the failure
>>>>>>>>>>>> scenarios above still apply even when a default-catalog is stored.
>>>>>>>>>>>> * The current spec also allows default-catalog to be null, in
>>>>>>>>>>>> which case it falls back to the view’s catalog, yet that catalog is
>>>>>>>>>>>> determined based on how the view is referenced in the query, which would be
>>>>>>>>>>>> considered non-deterministic based on the same criteria you shared.
>>>>>>>>>>>> * The only true form of determinism here is UUID-based
>>>>>>>>>>>> validation, which protects against silent drift in any resolution model.
>>>>>>>>>>>>
>>>>>>>>>>>> If the current model is considered deterministic, do you think
>>>>>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>>>>>> Or put another way: Would you be comfortable relying solely on
>>>>>>>>>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>>>>>>>>>> table, without UUID validation?
>>>>>>>>>>>>
>>>>>>>>>>>> +1 on involving other communities. I’m happy to help facilitate
>>>>>>>>>>>> a cross-community discussion if we aren’t able to reach a resolution here.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>>>>>>> Iceberg view spec. I am not convinced that the proposed change will make
>>>>>>>>>>>>> the table identifier resolution more clear and portable. The recommendation
>>>>>>>>>>>>> of using engines' current catalog and database can cause context dependent
>>>>>>>>>>>>> resolution results, which seems non-deterministic to me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>>>>>>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>>>>>>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>>>>>>>>>> more inputs from the broader community on this part.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> In Spark, I believe that the USE commands sets the current
>>>>>>>>>>>>>> catalog and namespace.  This affects both where the view is created and how
>>>>>>>>>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>>>>>>>>>> saving the current catalog and namespace into the view
>>>>>>>>>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > I think that's the lesser evil compared to Iceberg
>>>>>>>>>>>>>>> specifying how engines should resolve identifiers
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think this is also similar to the previous point. It is
>>>>>>>>>>>>>>> the other way around. Right now the spec dictates how to resolve (through
>>>>>>>>>>>>>>> employing a view-specific `default-catalog` field). The proposal is
>>>>>>>>>>>>>>> suggesting to get out of this space and let engines handle it similar
>>>>>>>>>>>>>>> to how they handle all identifiers.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> > I thought "default-catalog" could be set via the USE
>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Benny, I think this is a misconception or miscommunication.
>>>>>>>>>>>>>>>> The USE command has no impact on the `default-catalog` field. In fact, the
>>>>>>>>>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>>>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>>>>>>>>>> is not the case under the current spec.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <
>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> > there is no SQL construct today to explicitly set
>>>>>>>>>>>>>>>>> default-catalog
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I thought "default-catalog" could be set via the USE
>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I generally agree with Dan about requiring consistent
>>>>>>>>>>>>>>>>> catalog names.  I think that's the lesser evil compared to Iceberg
>>>>>>>>>>>>>>>>> specifying how engines should resolve identifiers.  Another thing to
>>>>>>>>>>>>>>>>> consider is that identifier resolution can be very expensive at query
>>>>>>>>>>>>>>>>> validation time if identifiers need to be looked up from a bunch of
>>>>>>>>>>>>>>>>> places.  Hopefully, it should be possible to define a view in such a way
>>>>>>>>>>>>>>>>> that identifiers can be resolved on the first try.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You're right that the proposal touches on two aspects,
>>>>>>>>>>>>>>>>>> and resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>>>>>>>>>> materialized views.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We may be able to converge on this storage format even if
>>>>>>>>>>>>>>>>>> we haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>>>>>>>>>> identifiers in this way.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm supportive of moving forward with consensus on the
>>>>>>>>>>>>>>>>>> identifier storage format. That said, we may continue to run into questions
>>>>>>>>>>>>>>>>>> related to resolution during implementation. For example: Should the
>>>>>>>>>>>>>>>>>> storage table identifier follow the same default-catalog and
>>>>>>>>>>>>>>>>>> default-namespace resolution behavior as other table references?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We can continue having the discussion if the current
>>>>>>>>>>>>>>>>>>> view spec needs to be changed or not. Based on the decision at a later
>>>>>>>>>>>>>>>>>>> point if required we can update the view resolution rule.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Rishabh
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Correction of typo: both engines seem to set
>>>>>>>>>>>>>>>>>>>> default-catalog to the view catalog if it is defined, or to null if the
>>>>>>>>>>>>>>>>>>>> view catalog is not defined.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I agree that catalog renaming is an environmental
>>>>>>>>>>>>>>>>>>>>> event, but it's a real one that happens frequently in practice.
>>>>>>>>>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate
>>>>>>>>>>>>>>>>>>>>> something as common as catalog renaming feels very restrictive, and could
>>>>>>>>>>>>>>>>>>>>> make the spec less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>>>>>>>>>> I’m sharing this from the perspective of a large data
>>>>>>>>>>>>>>>>>>>>> lake environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Further, it's worth noting that the table spec is
>>>>>>>>>>>>>>>>>>>>> resilient to catalog renaming, but the view spec is not. If we have an
>>>>>>>>>>>>>>>>>>>>> opportunity to make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>>>>>>>>>> Both specifications are deterministic in their
>>>>>>>>>>>>>>>>>>>>> definition, but one is more fragile to environmental changes than the
>>>>>>>>>>>>>>>>>>>>> other. Improving resilience does not sacrifice determinism. It simply makes
>>>>>>>>>>>>>>>>>>>>> views safer and more portable over time.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Separately, given that there is no SQL construct today
>>>>>>>>>>>>>>>>>>>>> to explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>>>>>>>>>> Today, both engines seem to set default-catalog to
>>>>>>>>>>>>>>>>>>>>> null if the view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>>>>>>>>>> What was the intended thought process behind this
>>>>>>>>>>>>>>>>>>>>> behavior?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > tables inside views remain reachable after a
>>>>>>>>>>>>>>>>>>>>>> catalog rename
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> This problem stems from the exact
>>>>>>>>>>>>>>>>>>>>>> environmental/configuration issue that we should not be trying to address.
>>>>>>>>>>>>>>>>>>>>>> I don't think we would expect references to survive a catalog rename.
>>>>>>>>>>>>>>>>>>>>>> That's not something covered by the spec and needs to be handled separately
>>>>>>>>>>>>>>>>>>>>>> as a platform-level migration specific to the affected environment.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The identifier resolution logic is clear and
>>>>>>>>>>>>>>>>>>>>>> deterministic.  It should not matter whether an engine resolves and encodes
>>>>>>>>>>>>>>>>>>>>>> the default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but
>>>>>>>>>>>>>>>>>>>>>> rather view behavior when you start altering the environment around it,
>>>>>>>>>>>>>>>>>>>>>> which isn't something we should be trying to define here.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just
>>>>>>>>>>>>>>>>>>>>>>> catalog naming consistency. The behavior around default-catalog itself
>>>>>>>>>>>>>>>>>>>>>>> introduces resolution inconsistencies even when catalog names are
>>>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside
>>>>>>>>>>>>>>>>>>>>>>> views remain reachable after a catalog rename. But if it is set to a
>>>>>>>>>>>>>>>>>>>>>>> non-null value, table references will break.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * default-catalog causes table references inside
>>>>>>>>>>>>>>>>>>>>>>> views to be early bound (i.e., bound at view creation time, especially when
>>>>>>>>>>>>>>>>>>>>>>> using a non-null value), while table references inside standalone queries
>>>>>>>>>>>>>>>>>>>>>>> are late bound (bound at query time). This creates inconsistencies when
>>>>>>>>>>>>>>>>>>>>>>> resolving the same table name inside and outside views, even within the
>>>>>>>>>>>>>>>>>>>>>>> same job.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from
>>>>>>>>>>>>>>>>>>>>>>> the spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * With view nesting (views depending on views),
>>>>>>>>>>>>>>>>>>>>>>> these inconsistencies amplify further, forcing users and engines to reason
>>>>>>>>>>>>>>>>>>>>>>> about catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to
>>>>>>>>>>>>>>>>>>>>>>> Iceberg with that model. Migrated Hive views will have to unfollow that
>>>>>>>>>>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> How would you suggest approaching the engine-level
>>>>>>>>>>>>>>>>>>>>>>> changes required to support the current default-catalog field?
>>>>>>>>>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities
>>>>>>>>>>>>>>>>>>>>>>> would align around having table resolution behave inconsistently between
>>>>>>>>>>>>>>>>>>>>>>> queries and views, or inconsistency between Iceberg and other types of
>>>>>>>>>>>>>>>>>>>>>>> views?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I would agree with Jan's summary of why
>>>>>>>>>>>>>>>>>>>>>>>> 'default-catalog' was introduced, but I think we need to step back and
>>>>>>>>>>>>>>>>>>>>>>>> align on what we are really attempting to support in the spec.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original
>>>>>>>>>>>>>>>>>>>>>>>> intent to address the catalog naming/configuration and would argue that we
>>>>>>>>>>>>>>>>>>>>>>>> shouldn't attempt to do that as part of the spec.  Inconsistently named
>>>>>>>>>>>>>>>>>>>>>>>> catalogs are a reality, but we should consider that a
>>>>>>>>>>>>>>>>>>>>>>>> configuration/environmental issue, not something to solve for in the spec.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> We should support and advocate for consistency in
>>>>>>>>>>>>>>>>>>>>>>>> catalog naming and define the spec along those lines.  The fact is that
>>>>>>>>>>>>>>>>>>>>>>>> with all of the recent work that's gone into making catalogs pluggable, it
>>>>>>>>>>>>>>>>>>>>>>>> makes more sense to just register catalog configuration with consistent
>>>>>>>>>>>>>>>>>>>>>>>> names (even if you have to duplicate the configuration for supporting
>>>>>>>>>>>>>>>>>>>>>>>> existing readers/writers).  I think it's better to provide a path toward
>>>>>>>>>>>>>>>>>>>>>>>> consistency than to normalize complicated schemes to workaround the issues
>>>>>>>>>>>>>>>>>>>>>>>> caused by environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the
>>>>>>>>>>>>>>>>>>>>>>>> late binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>> was that every query engine uses it to store its session default catalog at
>>>>>>>>>>>>>>>>>>>>>>>>> the time of creating the view. This way the view could be reused in another
>>>>>>>>>>>>>>>>>>>>>>>>> session. The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Generally we have different environments we want
>>>>>>>>>>>>>>>>>>>>>>>>> to support with the view spec:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> When the environment supports it, using consistent
>>>>>>>>>>>>>>>>>>>>>>>>> catalog names can have a great benefit for multi-catalog, multi-engine
>>>>>>>>>>>>>>>>>>>>>>>>> setups. With consistent catalog names, using the "default-catalog" field
>>>>>>>>>>>>>>>>>>>>>>>>> works without any issues.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> This can be the case when different query engines
>>>>>>>>>>>>>>>>>>>>>>>>> refer to the same physical catalog by different names. This often happens
>>>>>>>>>>>>>>>>>>>>>>>>> because different query engines use different strategies to setup the
>>>>>>>>>>>>>>>>>>>>>>>>> catalogs. If catalogs have inconsistent naming, using the "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>> field does not work because it is not guaranteed that the catalog name can
>>>>>>>>>>>>>>>>>>>>>>>>> be resolved with another engine. Using the "view catalog" as a fallback is
>>>>>>>>>>>>>>>>>>>>>>>>> a better solution for this use case, as it avoids catalog names altogether.
>>>>>>>>>>>>>>>>>>>>>>>>> It is however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> What do you think of introducing a view property
>>>>>>>>>>>>>>>>>>>>>>>>> that specifies if the "default-catalog" or the "view catalog" should be
>>>>>>>>>>>>>>>>>>>>>>>>> used? This way, you could use the "default-catalog" in environments where
>>>>>>>>>>>>>>>>>>>>>>>>> you can guarantee consistent naming, but you would be able to directly
>>>>>>>>>>>>>>>>>>>>>>>>> fallback to the "view-catalog" when you don't have consistent naming. The
>>>>>>>>>>>>>>>>>>>>>>>>> query engines could set the default for this view property at creation
>>>>>>>>>>>>>>>>>>>>>>>>> time. Spark for example could set it to automatically use the "view
>>>>>>>>>>>>>>>>>>>>>>>>> catalog".
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions
>>>>>>>>>>>>>>>>>>>>>>>>> and interpretation of the spec, I have summarized everything we discussed
>>>>>>>>>>>>>>>>>>>>>>>>> so far at the top of the proposal document (here
>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace
>>>>>>>>>>>>>>>>>>>>>>>>> fields from the view spec completely.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate
>>>>>>>>>>>>>>>>>>>>>>>>> view-level default catalogs or namespaces.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a
>>>>>>>>>>>>>>>>>>>>>>>>> catalog qualifier, engines should resolve it using the current engine
>>>>>>>>>>>>>>>>>>>>>>>>> catalog at query time.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata
>>>>>>>>>>>>>>>>>>>>>>>>> exactly as they appear in the view SQL text.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at
>>>>>>>>>>>>>>>>>>>>>>>>> creation, it should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the
>>>>>>>>>>>>>>>>>>>>>>>>>> confusion the fallback creates. Also just to be clear, at this point and
>>>>>>>>>>>>>>>>>>>>>>>>>> after clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior
>>>>>>>>>>>>>>>>>>>>>>>>>>> of "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I
>>>>>>>>>>>>>>>>>>>>>>>>>>> don't think we should design something that says first use this catalog,
>>>>>>>>>>>>>>>>>>>>>>>>>>> then that catalog.. or that catalog.  For example, resolving identifiers
>>>>>>>>>>>>>>>>>>>>>>>>>>> using default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise
>>>>>>>>>>>>>>>>>>>>>>>>>>> I see no value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Interoperability between different engine
>>>>>>>>>>>>>>>>>>>>>>>>>>> types is very hard due to dialect issues... so I think we should focus on
>>>>>>>>>>>>>>>>>>>>>>>>>>> supporting different clusters of the same engine type on a shared catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is
>>>>>>>>>>>>>>>>>>>>>>>>>>> along the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable
>>>>>>>>>>>>>>>>>>>>>>>>>>> identifier problem through the lens of catalog resolution but maybe the
>>>>>>>>>>>>>>>>>>>>>>>>>>> right approach is really about templating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>> identifiers in a view to span catalogs to support non-Iceberg tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>  Also, the default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark implementation conform to the spec? Do we need Spark SQL extensions
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and/or Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistencies I shared regarding many resolution methods not consistently
>>>>>>>>>>>>>>>>>>>>>>>>>>>> being followed in different scenarios (view vs child table resolution,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> query vs view resolution)? Note these occur when the default catalog is set
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to a non-null value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when `default-catalog` is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback should be the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matter what the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog, this is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog, catalog names
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be included in the reference name
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already. So no ambiguity
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> separate problem, which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can only recommend that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> usage for better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior to engine's session
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> than the current fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from this idea. Spark resolves table identifiers during view creation using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue for not conforming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined at view creation time. It would be circular: the view needs to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exist before its metadata (and hence default-catalog) can exist. This is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> visible in Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog setting would require extending SQL syntax and engine APIs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to promote it to a first-class view concept. This would be intrusive,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> non-intuitive, and realistically very difficult to standardize across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from this idea. Spark resolves table identifiers during view creation using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VIEW statement the query engine knowns which catalog the view is being
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> created in. So even though we typically use late binding to resolve the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the "view catalog" where the view is going to be created in. It can use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that catalog to resolve partial table identifiers if "default-catalog" is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not set.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> where partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AS SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do we validate that a table exists without a default catalog, or do we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always use the current session catalog?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. Just to clarify: when I referred to late binding as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “similar” to the proposal, I was acknowledging the distinction between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view-level and table-level resolution. But as you noted, both follow a late
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> binding model.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting question and a potential gap: if default-catalog is only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined at query time, how should resolution work during view creation? For
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, if we want to validate that the tables referenced in the view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exist, how can we do that when default-catalog isn't defined, since the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view hasn't been created or loaded yet?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish between the physical catalog and it's reference used in SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements. The important part is that the physical catalog of the view and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in it's definition stay consistent. You could create
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a view in a given physical catalog by referring to it as "catalogA", as in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> your first point. If you then, given a different setup, refer to the same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> physical catalog as "catalogB" in another session/environment, the behavior
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should still work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Late binding applies to the view catalog name and by extension to all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partial table references when no "default-catalog" is present. Resolving
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the view catalog name at query time is not opposed to storing the view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata in a catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you mean.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced in a SQL statement. That statement contains the information for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the query engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * FROM catalogB.namespace.view (after renaming catalogA to catalogB,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and keeping everything else the same), then catalogB becomes the view’s
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sounds to me like the catalog is resolved at query time, based on how the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view is referenced, not from any stored metadata. That would imply some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sort of a late binding behavior (similar to the proposal), as opposed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using some catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> current spec: The view is executed when it's being referenced in a SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement. That statement contains the information for the query engine to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolve the catalog of the view. The query engine then uses that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> information to fetch the view metadata from the catalog. It also needs to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporarily keep track of which catalog it used to fetch the view metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It can then use that information to resolve the table references in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views SQL definition in case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog can be referenced at execution time. As long as that's the case I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would assume the view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> because the current specification can lead to some unintuitive behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example for the following statement:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.sales.monthly_orders AS SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the view spec, because it has the "closure" property. Because of the fact
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that the "view catalog" has to be known when executing a view, all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> information required to resolve the table identifiers is contained in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view metadata (and the "view catalog"). I think that if you make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> identifier resolution dependent on external parameters, it hinders
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> portability.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point before going deeper:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (e.g., catalogA.namespace.table) can resolve to different objects depending
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> solely on engine-specific routing logic or catalog aliases. So determinism
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> isn’t guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you help clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog” stored, so that an engine knows to fall back to it if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog is null?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understood correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-namepace"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> points.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the current session setup is in my opinion the definition of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "non-determinism". You could be running the same query-engine and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog-setup on different days, with different default session catalogs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which is rather common), and would be getting different results.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view always produces the same results. The current behavior has some rough
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> edges in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback for "default-namespace"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> current implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the setup more robust. Which is great. However, having UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> still requires us to have a portable table identifier specification. Even
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if we have the UUIDs of the referenced tables from the view, there simply
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> isn't an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> setup and suiting table identifiers to obtain the table metadata. We can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the UUIDs to verify if we loaded the correct table. But this can only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be done after we used some identifier. Which means there is no way of using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for "default-catalog" because it is more deterministic in my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> opinion. And I think the current spec does a good job for multi-engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table identifier resolution. I see the UUID validation more of an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> additional hardening strategy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Liu <li...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have some concerns with resolving catalog names on the fly with query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine defined catalog names. This introduces some flexibility at first
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> glance, but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should store resolved table uuid in view metadata, as table/view renaming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may introduce errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> momentum and closing out the MV spec as well. I’m hoping we can proceed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a vote next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The proposal outlines a strategy for handling table identifiers in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view metadata, with the goal of ensuring correctness, portability, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine compatibility. It recommends resolving table identifiers at read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time (late binding) rather than creation time, and introduces UUID-based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation to maintain identity guarantees across engines, or sessions. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also revises how default-catalog and default-namespace are handled
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (defaulting both to the session context if not explicitly set) to better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> align with engine behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> addressed the comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that catalog names in the spec refer to logical catalogs—i.e., the first
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part of a three-part identifier. These correspond to Spark's DataSourceV2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, Trino connectors, and similar constructs. This is a level of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> abstraction above physical catalogs, which are not referenced or used in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the view spec. The reason is that table identifiers in the view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition/text itself refer to logical catalogs, not physical ones (since
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they interface directly with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun <su...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think view portability is a very important topic for us to continue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussing as it relies on many assumptions within the data ecosystem for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to function like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how this may impact the permission questions the engines will be asking,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and whether that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eduard Tudenhöfner <et...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> problem. I've added a few comments to get a better understanding of how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this will look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> our discussion on how to reference table identifiers from Iceberg metadata,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with a few community members to better understand how the current spec is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> being interpreted. Those conversations served as inputs to a new proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on how table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]. I look forward to your feedback and working together to move this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> forward so we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

Hi all,

I want to clarify the scope and intent of the proposal, as well as my
position in light of the recent discussion.

The proposal specifically focuses on how to represent table identifiers
when they appear in* Iceberg view metadata, not in table metadata*.

This includes cases such as:
(1) the storage table identifier for materialized views, and
(2) child tables or views when stored on the view side for lineage or
dependency tracking.

It does not necessarily apply to view children stored in table metadata for
freshness tracking. That’s a separate layer of the system, though some of
the same principles might apply.

The core motivation here is to promote consistency in table resolution
behavior. Introducing different resolution rules for different types of
identifiers even within the view itself (e.g., storage table vs. child
views) would only compound this inconsistency. That’s something I’d
strongly prefer to avoid.

Finally, this discussion raises an important point: even if we agree on a
common format for storing table identifiers, we still need to define how
they are resolved, particularly in the case of the storage table identifier.
If we document that resolution must go through default-catalog and
default-namespace, we risk institutionalizing a resolution rule that has
not yet reached consensus, especially if that behavior needs to be captured
in new documentation or specs. So while aligning on format is helpful, I’m
not yet sure it fully unblocks the spec without clearer agreement on the
associated resolution semantics.

Thanks,
Walaa.

On Tue, May 13, 2025 at 9:39 AM Steven Wu <st...@gmail.com> wrote:

> > 1. Are we talking about table identifiers in general or are we talking
> only about the storage table identifier or the source table identifiers?
>
> At least I thought Walaa's latest question is about the source table
> identifiers for the refresh-state metadata in storage table.
>
> > 2. When using optional fields for namespace and catalog, what are we
> using when these fields are not set? (Early vs late binding)
>
> I believe we are not talking about the table identifier resolution (early
> vs late) for the regular view spec anymore, which was the original
> intention of this thread.
>
> On Mon, May 12, 2025 at 11:56 PM Jan Kaul <ja...@mailbox.org.invalid>
> wrote:
>
>> I think it makes sense to distinguish between the identifiers for the
>> storage table and the source tables.
>>
>> *Storage table*: The table that stores the precomputed data of the
>> materialized view. The storage table identifier is part of the view
>> metadata and is used to access the precomputed data from the view.
>>
>> *Source table*(Base table): Is a table referenced in the materialized
>> view query. Its state is stored in the "refresh state"(lineage) as part of
>> the storage table metadata to determine the freshness of the materialized
>> view.
>>
>> And we need to clarify if the current discussion is about table
>> identifiers in general or if we're discussing either the storage table or
>> source table identifiers. The following is my opinion on these:
>>
>> 1. Storage table identifier
>>
>> It is functionally required to store a table identifier as part of the
>> view metadata to be able to fetch the storage table metadata from the
>> catalog. The catalog interface uses identifiers, so there is no alternative
>> here. The question remains if the identifier should use optional namespace
>> and catalog fields and what to do if these fields are not set. (I'm not
>> sure if Walaa was also proposing the add an UUID field)
>>
>> 2. Source table identifiers
>>
>> The source table identifiers can be obtained from parsing the SQL
>> definition of the view, so it's not strictly required to store them as part
>> of the metadata. At some point we agreed to use SQL parsing to obtain these
>> identifiers, because this fits to how the query engine is executing the
>> query. Moreover, this lets us use different representations for different
>> query engines. If the representation translation tool supports some kind of
>> mapping, we could have different "catalog aliases" with different engines.
>> We agreed to use UUIDs to reference source tables in the refresh-state.
>> The discussion steven mentioned, was about adding the source table
>> identifiers to the refresh-state(lineage) for readability, but not to be
>> used by engines.
>> The current design doesn't allow multiple representations for the
>> refresh-state(lineage). This makes it more difficult to solve the "catalog
>> alias" problem as part of the refresh-state. To avoid this problem and
>> since the table identifiers are not strictly required, I was advocating to
>> not store any source table identifiers in the refresh-state. In my opinion,
>> having one representation of source table identifiers in the refresh-state
>> is in conflict with having multiple representations for the SQL query,
>> where the identifiers occur.
>>
>> My current questions are:
>>
>> 1. Are we talking about table identifiers in general or are we talking
>> only about the storage table identifier or the source table identifiers?
>>
>> 2. When using optional fields for namespace and catalog, what are we
>> using when these fields are not set? (Early vs late binding)
>>
>> Thanks,
>>
>> Jan
>> On 5/12/25 18:23, Steven Wu wrote:
>>
>> Regarding the following representation for the storage table identifier:
>>
>> * catalog name: optional
>> * namespace: optional
>> * table name: required
>> * UUID: required
>> It is the same discussion as this github thread.
>> https://github.com/apache/iceberg/pull/11041/files#r1869998317
>>
>> I was in favor of adding them mostly from human readability when manually
>> inspecting the storage table to avoid a reverse lookup from table UUID to
>> table identifier.  Benny (and a few others) have expressed that it may add
>> more ambiguity (like this thread was discussing). To clarify, we only need
>> to include the identifier as they are defined in the SQL query (e.g.
>> catalog and namespace are optional).  I doubt it would add more ambiguity.
>> But they are certainly not functionally required. So I am not opinionated
>> on this one.
>>
>> On Mon, May 12, 2025 at 5:01 AM Jan Kaul <ja...@mailbox.org.invalid>
>> <ja...@mailbox.org.invalid> wrote:
>>
>>> I have a question regarding the following representation for the storage
>>> table identifier:
>>>
>>> * catalog name: optional
>>> * namespace: optional
>>> * table name: required
>>> * UUID: required
>>>
>>> Having optional fields as part of the identifier again raises the
>>> question of what to use if these optional fields are not set. And we again
>>> have the early vs late binding discussion. Where early binding would
>>> default to the view catalog and late binding would use the current session
>>> catalog/namespace.
>>>
>>> @Walaa: Which behavior did you have in mind?
>>>
>>> In my opinion, having the functionality of MVs depend on the current
>>> session context is not desirable. So I would argue against late binding.
>>>
>>> Best wishes,
>>>
>>> Jan
>>> On 5/8/25 19:59, Walaa Eldin Moustafa wrote:
>>>
>>> Let us continue here for the context and continuity of the conversation.
>>>
>>> On Thu, May 8, 2025 at 10:55 AM Steven Wu <st...@gmail.com> wrote:
>>>
>>>> For the storage table identifier discussion, can we move to Jan's PR on
>>>> materialized view spec?
>>>> https://github.com/apache/iceberg/pull/11041
>>>>
>>>>
>>>> On Thu, May 8, 2025 at 10:40 AM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> It looks like we may not reach alignment on the resolution method in
>>>>> the short term. That said, perhaps we can move forward by aligning on the
>>>>> format used to store table identifiers in metadata instead.
>>>>>
>>>>> The idea is that the same format could work under either resolution
>>>>> model, which would allow us to make progress independently of that debate.
>>>>>
>>>>> Is this storage format of the storage table identifier agreeable to
>>>>> you?
>>>>> * catalog name: optional
>>>>> * namespace: optional
>>>>> * table name: required
>>>>> * UUID: required
>>>>> With the condition that the components included in a table identifier
>>>>> in the metadata should match those used in the view text for that table.
>>>>>
>>>>> The same structure above could apply to storing child tables or view
>>>>> identifiers as part of lineage, if and when that’s needed.
>>>>>
>>>>> For the longer term resolution model, it might make sense to initiate
>>>>> a cross-community discussion.
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>> On Thu, May 8, 2025 at 9:07 AM Steven Wu <st...@gmail.com> wrote:
>>>>>
>>>>>> If a table referenced by the view SQL query is deleted and recreated,
>>>>>> is the regular view still valid? If yes, it won't be correct to use
>>>>>> resolved UUID to validate the name reference. On the materialized view
>>>>>> side, we are aligned that UUID is necessary to detect this scenario and
>>>>>> mark the refresh state "stale" if a referenced table is re-created.
>>>>>>
>>>>>> > esolution using default-catalog and default-namespace does not
>>>>>> provide full determinism,
>>>>>>
>>>>>> With the new proposal, the catalog and namespace depends on the
>>>>>> engine's current context. if the user changes the current context (catalog
>>>>>> and database) via SET/USE , the view SQL query may resolve to a different
>>>>>> table. That is probably a bigger problem.
>>>>>>
>>>>>> With the current view spec, the intention is set in the view creation
>>>>>> time. I would rather have the view producer set the explicit intention
>>>>>> (than the implicit dependency on the engine context at view consumption
>>>>>> phase).
>>>>>>
>>>>>> Also the view SQL can still reference a table with a catalog name
>>>>>> component.  Hence the catalog naming still matters.The new proposal won't
>>>>>> really solve the naming problem.
>>>>>>
>>>>>> On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Dan,
>>>>>>>
>>>>>>> Thanks for the clarification! I agree that UUIDs should not be
>>>>>>> stored in the SQL definition of the view.
>>>>>>>
>>>>>>> Just to clarify: the proposal isn't about modifying the view
>>>>>>> definition itself, but rather about how table identifiers are represented
>>>>>>> in Iceberg view metadata, which naturally depends on how those identifiers
>>>>>>> are resolved.
>>>>>>>
>>>>>>> For example, in order to store the "storage table identifier" in the
>>>>>>> metadata:
>>>>>>> * Should we apply the same resolution rules based on default-catalog
>>>>>>> and default-namespace?
>>>>>>> * And if that resolution method is considered deterministic enough,
>>>>>>> would you be comfortable not storing the UUID in the storage table
>>>>>>> identifier metadata?
>>>>>>>
>>>>>>> The same questions apply when storing child views or tables as part
>>>>>>> of the view’s lineage metadata. It would be helpful to clarify the
>>>>>>> resolution model and whether UUIDs should be part of that for validation or
>>>>>>> drift detection.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Walaa.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I don't think we want to include the resolved table UUIDs in the
>>>>>>>> view definition, but rather in the storage table state.  You can still
>>>>>>>> resolve whether those drift at some point, but I don't feel like it's a
>>>>>>>> good idea to capture data in the view that we may allow to drift if there
>>>>>>>> isn't any requirement that they match.  This also aligns with the
>>>>>>>> identifier resolution being late binding.
>>>>>>>>
>>>>>>>> -Dan
>>>>>>>>
>>>>>>>> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Steven! So would you agree that resolution using
>>>>>>>>> default-catalog and default-namespace does not provide full determinism,
>>>>>>>>> and requires a supporting safety mechanism?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Walaa.
>>>>>>>>>
>>>>>>>>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> > If the current model is considered deterministic, do you think
>>>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>>>>
>>>>>>>>>> I am fine with storing UUIDs for table identifiers in the view.
>>>>>>>>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>>>>>>>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>>>>>>>>> fail the query if mismatch.
>>>>>>>>>>
>>>>>>>>>> The UUID change doesn't really change the table identifier
>>>>>>>>>> resolution rule though. It is more of a safety protection.
>>>>>>>>>>
>>>>>>>>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Steven,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>
>>>>>>>>>>> > I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>>>>> Iceberg view spec.
>>>>>>>>>>>
>>>>>>>>>>> To clarify, I don't believe the proposal is trying to solve
>>>>>>>>>>> catalog naming. What it’s doing is simply this:
>>>>>>>>>>>
>>>>>>>>>>> * Proposing that table names inside views resolve the same way
>>>>>>>>>>> as they do elsewhere (e.g., queries).
>>>>>>>>>>> * Adopting a model that is already widely used and supported in
>>>>>>>>>>> the existing ecosystem, which allows for:
>>>>>>>>>>>     -- Renaming catalog aliases
>>>>>>>>>>>     -- Swapping catalog implementations behind consistent names
>>>>>>>>>>>     -- Having different default catalog names across engines
>>>>>>>>>>> that still point to the same underlying tables
>>>>>>>>>>>
>>>>>>>>>>> These are common patterns in production data lakes. Saying
>>>>>>>>>>> Iceberg views cannot operate in those environments feels unrealistic. In
>>>>>>>>>>> practice, it means the spec breaks down in situations that users encounter
>>>>>>>>>>> regularly.
>>>>>>>>>>>
>>>>>>>>>>> > The recommendation of using engines’ current catalog and
>>>>>>>>>>> database can cause context-dependent resolution results.
>>>>>>>>>>>
>>>>>>>>>>> * As noted in the doc and earlier replies, fixing a catalog name
>>>>>>>>>>> doesn’t actually guarantee determinism either. All the failure scenarios
>>>>>>>>>>> above still apply even when a default-catalog is stored.
>>>>>>>>>>> * The current spec also allows default-catalog to be null, in
>>>>>>>>>>> which case it falls back to the view’s catalog, yet that catalog is
>>>>>>>>>>> determined based on how the view is referenced in the query, which would be
>>>>>>>>>>> considered non-deterministic based on the same criteria you shared.
>>>>>>>>>>> * The only true form of determinism here is UUID-based
>>>>>>>>>>> validation, which protects against silent drift in any resolution model.
>>>>>>>>>>>
>>>>>>>>>>> If the current model is considered deterministic, do you think
>>>>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>>>>> Or put another way: Would you be comfortable relying solely on
>>>>>>>>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>>>>>>>>> table, without UUID validation?
>>>>>>>>>>>
>>>>>>>>>>> +1 on involving other communities. I’m happy to help facilitate
>>>>>>>>>>> a cross-community discussion if we aren’t able to reach a resolution here.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Walaa.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>>>>>> Iceberg view spec. I am not convinced that the proposed change will make
>>>>>>>>>>>> the table identifier resolution more clear and portable. The recommendation
>>>>>>>>>>>> of using engines' current catalog and database can cause context dependent
>>>>>>>>>>>> resolution results, which seems non-deterministic to me.
>>>>>>>>>>>>
>>>>>>>>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>>>>>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>>>>>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>>>>>>>>> more inputs from the broader community on this part.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> In Spark, I believe that the USE commands sets the current
>>>>>>>>>>>>> catalog and namespace.  This affects both where the view is created and how
>>>>>>>>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>>>>>>>>> saving the current catalog and namespace into the view
>>>>>>>>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > I think that's the lesser evil compared to Iceberg
>>>>>>>>>>>>>> specifying how engines should resolve identifiers
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think this is also similar to the previous point. It is the
>>>>>>>>>>>>>> other way around. Right now the spec dictates how to resolve (through
>>>>>>>>>>>>>> employing a view-specific `default-catalog` field). The proposal is
>>>>>>>>>>>>>> suggesting to get out of this space and let engines handle it similar
>>>>>>>>>>>>>> to how they handle all identifiers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > I thought "default-catalog" could be set via the USE
>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Benny, I think this is a misconception or miscommunication.
>>>>>>>>>>>>>>> The USE command has no impact on the `default-catalog` field. In fact, the
>>>>>>>>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>>>>>>>>> is not the case under the current spec.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> > there is no SQL construct today to explicitly set
>>>>>>>>>>>>>>>> default-catalog
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I thought "default-catalog" could be set via the USE
>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I generally agree with Dan about requiring consistent
>>>>>>>>>>>>>>>> catalog names.  I think that's the lesser evil compared to Iceberg
>>>>>>>>>>>>>>>> specifying how engines should resolve identifiers.  Another thing to
>>>>>>>>>>>>>>>> consider is that identifier resolution can be very expensive at query
>>>>>>>>>>>>>>>> validation time if identifiers need to be looked up from a bunch of
>>>>>>>>>>>>>>>> places.  Hopefully, it should be possible to define a view in such a way
>>>>>>>>>>>>>>>> that identifiers can be resolved on the first try.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>>>>>>>>> materialized views.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We may be able to converge on this storage format even if
>>>>>>>>>>>>>>>>> we haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>>>>>>>>> identifiers in this way.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm supportive of moving forward with consensus on the
>>>>>>>>>>>>>>>>> identifier storage format. That said, we may continue to run into questions
>>>>>>>>>>>>>>>>> related to resolution during implementation. For example: Should the
>>>>>>>>>>>>>>>>> storage table identifier follow the same default-catalog and
>>>>>>>>>>>>>>>>> default-namespace resolution behavior as other table references?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We can continue having the discussion if the current view
>>>>>>>>>>>>>>>>>> spec needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Rishabh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Correction of typo: both engines seem to set
>>>>>>>>>>>>>>>>>>> default-catalog to the view catalog if it is defined, or to null if the
>>>>>>>>>>>>>>>>>>> view catalog is not defined.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I agree that catalog renaming is an environmental
>>>>>>>>>>>>>>>>>>>> event, but it's a real one that happens frequently in practice.
>>>>>>>>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate
>>>>>>>>>>>>>>>>>>>> something as common as catalog renaming feels very restrictive, and could
>>>>>>>>>>>>>>>>>>>> make the spec less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>>>>>>>>> I’m sharing this from the perspective of a large data
>>>>>>>>>>>>>>>>>>>> lake environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Further, it's worth noting that the table spec is
>>>>>>>>>>>>>>>>>>>> resilient to catalog renaming, but the view spec is not. If we have an
>>>>>>>>>>>>>>>>>>>> opportunity to make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>>>>>>>>> Both specifications are deterministic in their
>>>>>>>>>>>>>>>>>>>> definition, but one is more fragile to environmental changes than the
>>>>>>>>>>>>>>>>>>>> other. Improving resilience does not sacrifice determinism. It simply makes
>>>>>>>>>>>>>>>>>>>> views safer and more portable over time.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Separately, given that there is no SQL construct today
>>>>>>>>>>>>>>>>>>>> to explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>>>>>>>>> Today, both engines seem to set default-catalog to null
>>>>>>>>>>>>>>>>>>>> if the view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>>>>>>>>> What was the intended thought process behind this
>>>>>>>>>>>>>>>>>>>> behavior?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > tables inside views remain reachable after a catalog
>>>>>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This problem stems from the exact
>>>>>>>>>>>>>>>>>>>>> environmental/configuration issue that we should not be trying to address.
>>>>>>>>>>>>>>>>>>>>> I don't think we would expect references to survive a catalog rename.
>>>>>>>>>>>>>>>>>>>>> That's not something covered by the spec and needs to be handled separately
>>>>>>>>>>>>>>>>>>>>> as a platform-level migration specific to the affected environment.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The identifier resolution logic is clear and
>>>>>>>>>>>>>>>>>>>>> deterministic.  It should not matter whether an engine resolves and encodes
>>>>>>>>>>>>>>>>>>>>> the default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but
>>>>>>>>>>>>>>>>>>>>> rather view behavior when you start altering the environment around it,
>>>>>>>>>>>>>>>>>>>>> which isn't something we should be trying to define here.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just
>>>>>>>>>>>>>>>>>>>>>> catalog naming consistency. The behavior around default-catalog itself
>>>>>>>>>>>>>>>>>>>>>> introduces resolution inconsistencies even when catalog names are
>>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside
>>>>>>>>>>>>>>>>>>>>>> views remain reachable after a catalog rename. But if it is set to a
>>>>>>>>>>>>>>>>>>>>>> non-null value, table references will break.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * default-catalog causes table references inside
>>>>>>>>>>>>>>>>>>>>>> views to be early bound (i.e., bound at view creation time, especially when
>>>>>>>>>>>>>>>>>>>>>> using a non-null value), while table references inside standalone queries
>>>>>>>>>>>>>>>>>>>>>> are late bound (bound at query time). This creates inconsistencies when
>>>>>>>>>>>>>>>>>>>>>> resolving the same table name inside and outside views, even within the
>>>>>>>>>>>>>>>>>>>>>> same job.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from
>>>>>>>>>>>>>>>>>>>>>> the spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to
>>>>>>>>>>>>>>>>>>>>>> Iceberg with that model. Migrated Hive views will have to unfollow that
>>>>>>>>>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> How would you suggest approaching the engine-level
>>>>>>>>>>>>>>>>>>>>>> changes required to support the current default-catalog field?
>>>>>>>>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities
>>>>>>>>>>>>>>>>>>>>>> would align around having table resolution behave inconsistently between
>>>>>>>>>>>>>>>>>>>>>> queries and views, or inconsistency between Iceberg and other types of
>>>>>>>>>>>>>>>>>>>>>> views?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I would agree with Jan's summary of why
>>>>>>>>>>>>>>>>>>>>>>> 'default-catalog' was introduced, but I think we need to step back and
>>>>>>>>>>>>>>>>>>>>>>> align on what we are really attempting to support in the spec.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original
>>>>>>>>>>>>>>>>>>>>>>> intent to address the catalog naming/configuration and would argue that we
>>>>>>>>>>>>>>>>>>>>>>> shouldn't attempt to do that as part of the spec.  Inconsistently named
>>>>>>>>>>>>>>>>>>>>>>> catalogs are a reality, but we should consider that a
>>>>>>>>>>>>>>>>>>>>>>> configuration/environmental issue, not something to solve for in the spec.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> We should support and advocate for consistency in
>>>>>>>>>>>>>>>>>>>>>>> catalog naming and define the spec along those lines.  The fact is that
>>>>>>>>>>>>>>>>>>>>>>> with all of the recent work that's gone into making catalogs pluggable, it
>>>>>>>>>>>>>>>>>>>>>>> makes more sense to just register catalog configuration with consistent
>>>>>>>>>>>>>>>>>>>>>>> names (even if you have to duplicate the configuration for supporting
>>>>>>>>>>>>>>>>>>>>>>> existing readers/writers).  I think it's better to provide a path toward
>>>>>>>>>>>>>>>>>>>>>>> consistency than to normalize complicated schemes to workaround the issues
>>>>>>>>>>>>>>>>>>>>>>> caused by environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the
>>>>>>>>>>>>>>>>>>>>>>> late binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>> was that every query engine uses it to store its session default catalog at
>>>>>>>>>>>>>>>>>>>>>>>> the time of creating the view. This way the view could be reused in another
>>>>>>>>>>>>>>>>>>>>>>>> session. The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Generally we have different environments we want to
>>>>>>>>>>>>>>>>>>>>>>>> support with the view spec:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> When the environment supports it, using consistent
>>>>>>>>>>>>>>>>>>>>>>>> catalog names can have a great benefit for multi-catalog, multi-engine
>>>>>>>>>>>>>>>>>>>>>>>> setups. With consistent catalog names, using the "default-catalog" field
>>>>>>>>>>>>>>>>>>>>>>>> works without any issues.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> This can be the case when different query engines
>>>>>>>>>>>>>>>>>>>>>>>> refer to the same physical catalog by different names. This often happens
>>>>>>>>>>>>>>>>>>>>>>>> because different query engines use different strategies to setup the
>>>>>>>>>>>>>>>>>>>>>>>> catalogs. If catalogs have inconsistent naming, using the "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>> field does not work because it is not guaranteed that the catalog name can
>>>>>>>>>>>>>>>>>>>>>>>> be resolved with another engine. Using the "view catalog" as a fallback is
>>>>>>>>>>>>>>>>>>>>>>>> a better solution for this use case, as it avoids catalog names altogether.
>>>>>>>>>>>>>>>>>>>>>>>> It is however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> What do you think of introducing a view property
>>>>>>>>>>>>>>>>>>>>>>>> that specifies if the "default-catalog" or the "view catalog" should be
>>>>>>>>>>>>>>>>>>>>>>>> used? This way, you could use the "default-catalog" in environments where
>>>>>>>>>>>>>>>>>>>>>>>> you can guarantee consistent naming, but you would be able to directly
>>>>>>>>>>>>>>>>>>>>>>>> fallback to the "view-catalog" when you don't have consistent naming. The
>>>>>>>>>>>>>>>>>>>>>>>> query engines could set the default for this view property at creation
>>>>>>>>>>>>>>>>>>>>>>>> time. Spark for example could set it to automatically use the "view
>>>>>>>>>>>>>>>>>>>>>>>> catalog".
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions
>>>>>>>>>>>>>>>>>>>>>>>> and interpretation of the spec, I have summarized everything we discussed
>>>>>>>>>>>>>>>>>>>>>>>> so far at the top of the proposal document (here
>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace
>>>>>>>>>>>>>>>>>>>>>>>> fields from the view spec completely.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate
>>>>>>>>>>>>>>>>>>>>>>>> view-level default catalogs or namespaces.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a
>>>>>>>>>>>>>>>>>>>>>>>> catalog qualifier, engines should resolve it using the current engine
>>>>>>>>>>>>>>>>>>>>>>>> catalog at query time.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata
>>>>>>>>>>>>>>>>>>>>>>>> exactly as they appear in the view SQL text.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at
>>>>>>>>>>>>>>>>>>>>>>>> creation, it should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the
>>>>>>>>>>>>>>>>>>>>>>>>> confusion the fallback creates. Also just to be clear, at this point and
>>>>>>>>>>>>>>>>>>>>>>>>> after clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior
>>>>>>>>>>>>>>>>>>>>>>>>>> of "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I
>>>>>>>>>>>>>>>>>>>>>>>>>> don't think we should design something that says first use this catalog,
>>>>>>>>>>>>>>>>>>>>>>>>>> then that catalog.. or that catalog.  For example, resolving identifiers
>>>>>>>>>>>>>>>>>>>>>>>>>> using default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I
>>>>>>>>>>>>>>>>>>>>>>>>>> see no value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>>>>>>>>> - Interoperability between different engine types
>>>>>>>>>>>>>>>>>>>>>>>>>> is very hard due to dialect issues... so I think we should focus on
>>>>>>>>>>>>>>>>>>>>>>>>>> supporting different clusters of the same engine type on a shared catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>> For example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is
>>>>>>>>>>>>>>>>>>>>>>>>>> along the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable
>>>>>>>>>>>>>>>>>>>>>>>>>> identifier problem through the lens of catalog resolution but maybe the
>>>>>>>>>>>>>>>>>>>>>>>>>> right approach is really about templating.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow
>>>>>>>>>>>>>>>>>>>>>>>>>> identifiers in a view to span catalogs to support non-Iceberg tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>  Also, the default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the
>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistencies I shared regarding many resolution methods not consistently
>>>>>>>>>>>>>>>>>>>>>>>>>>> being followed in different scenarios (view vs child table resolution,
>>>>>>>>>>>>>>>>>>>>>>>>>>> query vs view resolution)? Note these occur when the default catalog is set
>>>>>>>>>>>>>>>>>>>>>>>>>>> to a non-null value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior
>>>>>>>>>>>>>>>>>>>>>>>>>>>> when `default-catalog` is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback should be the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really
>>>>>>>>>>>>>>>>>>>>>>>>>>>> matter what the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog, this is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog, catalog names
>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be included in the reference name
>>>>>>>>>>>>>>>>>>>>>>>>>>>> already. So no ambiguity
>>>>>>>>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> separate problem, which
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can
>>>>>>>>>>>>>>>>>>>>>>>>>>>> only recommend that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across
>>>>>>>>>>>>>>>>>>>>>>>>>>>> usage for better
>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior to engine's session
>>>>>>>>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better
>>>>>>>>>>>>>>>>>>>>>>>>>>>> than the current fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from this idea. Spark resolves table identifiers during view creation using
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation
>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue for not conforming
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> want to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>> at view creation time. It would be circular: the view needs to exist before
>>>>>>>>>>>>>>>>>>>>>>>>>>>> its metadata (and hence default-catalog) can exist. This is visible in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>> setting would require extending SQL syntax and engine APIs to promote it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from this idea. Spark resolves table identifiers during view creation using
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>> VIEW statement the query engine knowns which catalog the view is being
>>>>>>>>>>>>>>>>>>>>>>>>>>>> created in. So even though we typically use late binding to resolve the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" where the view is going to be created in. It can use
>>>>>>>>>>>>>>>>>>>>>>>>>>>> that catalog to resolve partial table identifiers if "default-catalog" is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> not set.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> where partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do we validate that a table exists without a default catalog, or do we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> always use the current session catalog?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same
>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. Just to clarify: when I referred to late binding as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> “similar” to the proposal, I was acknowledging the distinction between
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view-level and table-level resolution. But as you noted, both follow a late
>>>>>>>>>>>>>>>>>>>>>>>>>>>> binding model.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish between the physical catalog and it's reference used in SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements. The important part is that the physical catalog of the view and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in it's definition stay consistent. You could create
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a view in a given physical catalog by referring to it as "catalogA", as in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> your first point. If you then, given a different setup, refer to the same
>>>>>>>>>>>>>>>>>>>>>>>>>>>> physical catalog as "catalogB" in another session/environment, the behavior
>>>>>>>>>>>>>>>>>>>>>>>>>>>> should still work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Late binding applies to the view catalog name and by extension to all
>>>>>>>>>>>>>>>>>>>>>>>>>>>> partial table references when no "default-catalog" is present. Resolving
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the view catalog name at query time is not opposed to storing the view
>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata in a catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what
>>>>>>>>>>>>>>>>>>>>>>>>>>>> you mean.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being
>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced in a SQL statement. That statement contains the information for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the query engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * FROM catalogB.namespace.view (after renaming catalogA to catalogB,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and keeping everything else the same), then catalogB becomes the view’s
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> sounds to me like the catalog is resolved at query time, based on how the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view is referenced, not from any stored metadata. That would imply some
>>>>>>>>>>>>>>>>>>>>>>>>>>>> sort of a late binding behavior (similar to the proposal), as opposed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> using some catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> current spec: The view is executed when it's being referenced in a SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement. That statement contains the information for the query engine to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolve the catalog of the view. The query engine then uses that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> information to fetch the view metadata from the catalog. It also needs to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporarily keep track of which catalog it used to fetch the view metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It can then use that information to resolve the table references in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> views SQL definition in case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>> can be referenced at execution time. As long as that's the case I would
>>>>>>>>>>>>>>>>>>>>>>>>>>>> assume the view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable
>>>>>>>>>>>>>>>>>>>>>>>>>>>> because the current specification can lead to some unintuitive behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example for the following statement:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AS SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view spec, because it has the "closure" property. Because of the fact that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" has to be known when executing a view, all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> information required to resolve the table identifiers is contained in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view metadata (and the "view catalog"). I think that if you make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> identifier resolution dependent on external parameters, it hinders
>>>>>>>>>>>>>>>>>>>>>>>>>>>> portability.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key
>>>>>>>>>>>>>>>>>>>>>>>>>>>> point before going deeper:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (e.g., catalogA.namespace.table) can resolve to different objects depending
>>>>>>>>>>>>>>>>>>>>>>>>>>>> solely on engine-specific routing logic or catalog aliases. So determinism
>>>>>>>>>>>>>>>>>>>>>>>>>>>> isn’t guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could
>>>>>>>>>>>>>>>>>>>>>>>>>>>> you help clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog” stored, so that an engine knows to fall back to it if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog is null?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> understood correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-namepace"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these
>>>>>>>>>>>>>>>>>>>>>>>>>>>> points.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the current session setup is in my opinion the definition of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "non-determinism". You could be running the same query-engine and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog-setup on different days, with different default session catalogs
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which is rather common), and would be getting different results.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view always produces the same results. The current behavior has some rough
>>>>>>>>>>>>>>>>>>>>>>>>>>>> edges in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-namespace"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> current implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the setup more robust. Which is great. However, having UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>> still requires us to have a portable table identifier specification. Even
>>>>>>>>>>>>>>>>>>>>>>>>>>>> if we have the UUIDs of the referenced tables from the view, there simply
>>>>>>>>>>>>>>>>>>>>>>>>>>>> isn't an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>> setup and suiting table identifiers to obtain the table metadata. We can
>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the UUIDs to verify if we loaded the correct table. But this can only
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be done after we used some identifier. Which means there is no way of using
>>>>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current
>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for "default-catalog" because it is more deterministic in my
>>>>>>>>>>>>>>>>>>>>>>>>>>>> opinion. And I think the current spec does a good job for multi-engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>> table identifier resolution. I see the UUID validation more of an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> additional hardening strategy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Liu <li...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> have some concerns with resolving catalog names on the fly with query
>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine defined catalog names. This introduces some flexibility at first
>>>>>>>>>>>>>>>>>>>>>>>>>>>> glance, but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> should store resolved table uuid in view metadata, as table/view renaming
>>>>>>>>>>>>>>>>>>>>>>>>>>>> may introduce errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> momentum and closing out the MV spec as well. I’m hoping we can proceed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a vote next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The proposal outlines a strategy for handling table identifiers in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view metadata, with the goal of ensuring correctness, portability, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine compatibility. It recommends resolving table identifiers at read
>>>>>>>>>>>>>>>>>>>>>>>>>>>> time (late binding) rather than creation time, and introduces UUID-based
>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation to maintain identity guarantees across engines, or sessions. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>> also revises how default-catalog and default-namespace are handled
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (defaulting both to the session context if not explicitly set) to better
>>>>>>>>>>>>>>>>>>>>>>>>>>>> align with engine behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have
>>>>>>>>>>>>>>>>>>>>>>>>>>>> addressed the comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog names in the spec refer to logical catalogs—i.e., the first part of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun <su...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> think view portability is a very important topic for us to continue
>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussing as it relies on many assumptions within the data ecosystem for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to function like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this may impact the permission questions the engines will be asking, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eduard Tudenhöfner <et...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> problem. I've added a few comments to get a better understanding of how
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this will look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline
>>>>>>>>>>>>>>>>>>>>>>>>>>>> with a few community members to better understand how the current spec is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> being interpreted. Those conversations served as inputs to a new proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>> on how table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]. I look forward to your feedback and working together to move this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> forward so we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Steven Wu <st...@gmail.com>]

> 1. Are we talking about table identifiers in general or are we talking
only about the storage table identifier or the source table identifiers?

At least I thought Walaa's latest question is about the source table
identifiers for the refresh-state metadata in storage table.

> 2. When using optional fields for namespace and catalog, what are we
using when these fields are not set? (Early vs late binding)

I believe we are not talking about the table identifier resolution (early
vs late) for the regular view spec anymore, which was the original
intention of this thread.

On Mon, May 12, 2025 at 11:56 PM Jan Kaul <ja...@mailbox.org.invalid>
wrote:

> I think it makes sense to distinguish between the identifiers for the
> storage table and the source tables.
>
> *Storage table*: The table that stores the precomputed data of the
> materialized view. The storage table identifier is part of the view
> metadata and is used to access the precomputed data from the view.
>
> *Source table*(Base table): Is a table referenced in the materialized
> view query. Its state is stored in the "refresh state"(lineage) as part of
> the storage table metadata to determine the freshness of the materialized
> view.
>
> And we need to clarify if the current discussion is about table
> identifiers in general or if we're discussing either the storage table or
> source table identifiers. The following is my opinion on these:
>
> 1. Storage table identifier
>
> It is functionally required to store a table identifier as part of the
> view metadata to be able to fetch the storage table metadata from the
> catalog. The catalog interface uses identifiers, so there is no alternative
> here. The question remains if the identifier should use optional namespace
> and catalog fields and what to do if these fields are not set. (I'm not
> sure if Walaa was also proposing the add an UUID field)
>
> 2. Source table identifiers
>
> The source table identifiers can be obtained from parsing the SQL
> definition of the view, so it's not strictly required to store them as part
> of the metadata. At some point we agreed to use SQL parsing to obtain these
> identifiers, because this fits to how the query engine is executing the
> query. Moreover, this lets us use different representations for different
> query engines. If the representation translation tool supports some kind of
> mapping, we could have different "catalog aliases" with different engines.
> We agreed to use UUIDs to reference source tables in the refresh-state.
> The discussion steven mentioned, was about adding the source table
> identifiers to the refresh-state(lineage) for readability, but not to be
> used by engines.
> The current design doesn't allow multiple representations for the
> refresh-state(lineage). This makes it more difficult to solve the "catalog
> alias" problem as part of the refresh-state. To avoid this problem and
> since the table identifiers are not strictly required, I was advocating to
> not store any source table identifiers in the refresh-state. In my opinion,
> having one representation of source table identifiers in the refresh-state
> is in conflict with having multiple representations for the SQL query,
> where the identifiers occur.
>
> My current questions are:
>
> 1. Are we talking about table identifiers in general or are we talking
> only about the storage table identifier or the source table identifiers?
>
> 2. When using optional fields for namespace and catalog, what are we using
> when these fields are not set? (Early vs late binding)
>
> Thanks,
>
> Jan
> On 5/12/25 18:23, Steven Wu wrote:
>
> Regarding the following representation for the storage table identifier:
>
> * catalog name: optional
> * namespace: optional
> * table name: required
> * UUID: required
> It is the same discussion as this github thread.
> https://github.com/apache/iceberg/pull/11041/files#r1869998317
>
> I was in favor of adding them mostly from human readability when manually
> inspecting the storage table to avoid a reverse lookup from table UUID to
> table identifier.  Benny (and a few others) have expressed that it may add
> more ambiguity (like this thread was discussing). To clarify, we only need
> to include the identifier as they are defined in the SQL query (e.g.
> catalog and namespace are optional).  I doubt it would add more ambiguity.
> But they are certainly not functionally required. So I am not opinionated
> on this one.
>
> On Mon, May 12, 2025 at 5:01 AM Jan Kaul <ja...@mailbox.org.invalid>
> <ja...@mailbox.org.invalid> wrote:
>
>> I have a question regarding the following representation for the storage
>> table identifier:
>>
>> * catalog name: optional
>> * namespace: optional
>> * table name: required
>> * UUID: required
>>
>> Having optional fields as part of the identifier again raises the
>> question of what to use if these optional fields are not set. And we again
>> have the early vs late binding discussion. Where early binding would
>> default to the view catalog and late binding would use the current session
>> catalog/namespace.
>>
>> @Walaa: Which behavior did you have in mind?
>>
>> In my opinion, having the functionality of MVs depend on the current
>> session context is not desirable. So I would argue against late binding.
>>
>> Best wishes,
>>
>> Jan
>> On 5/8/25 19:59, Walaa Eldin Moustafa wrote:
>>
>> Let us continue here for the context and continuity of the conversation.
>>
>> On Thu, May 8, 2025 at 10:55 AM Steven Wu <st...@gmail.com> wrote:
>>
>>> For the storage table identifier discussion, can we move to Jan's PR on
>>> materialized view spec?
>>> https://github.com/apache/iceberg/pull/11041
>>>
>>>
>>> On Thu, May 8, 2025 at 10:40 AM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> It looks like we may not reach alignment on the resolution method in
>>>> the short term. That said, perhaps we can move forward by aligning on the
>>>> format used to store table identifiers in metadata instead.
>>>>
>>>> The idea is that the same format could work under either resolution
>>>> model, which would allow us to make progress independently of that debate.
>>>>
>>>> Is this storage format of the storage table identifier agreeable to you?
>>>> * catalog name: optional
>>>> * namespace: optional
>>>> * table name: required
>>>> * UUID: required
>>>> With the condition that the components included in a table identifier
>>>> in the metadata should match those used in the view text for that table.
>>>>
>>>> The same structure above could apply to storing child tables or view
>>>> identifiers as part of lineage, if and when that’s needed.
>>>>
>>>> For the longer term resolution model, it might make sense to initiate a
>>>> cross-community discussion.
>>>>
>>>> Thanks,
>>>> Walaa.
>>>>
>>>> On Thu, May 8, 2025 at 9:07 AM Steven Wu <st...@gmail.com> wrote:
>>>>
>>>>> If a table referenced by the view SQL query is deleted and recreated,
>>>>> is the regular view still valid? If yes, it won't be correct to use
>>>>> resolved UUID to validate the name reference. On the materialized view
>>>>> side, we are aligned that UUID is necessary to detect this scenario and
>>>>> mark the refresh state "stale" if a referenced table is re-created.
>>>>>
>>>>> > esolution using default-catalog and default-namespace does not
>>>>> provide full determinism,
>>>>>
>>>>> With the new proposal, the catalog and namespace depends on the
>>>>> engine's current context. if the user changes the current context (catalog
>>>>> and database) via SET/USE , the view SQL query may resolve to a different
>>>>> table. That is probably a bigger problem.
>>>>>
>>>>> With the current view spec, the intention is set in the view creation
>>>>> time. I would rather have the view producer set the explicit intention
>>>>> (than the implicit dependency on the engine context at view consumption
>>>>> phase).
>>>>>
>>>>> Also the view SQL can still reference a table with a catalog name
>>>>> component.  Hence the catalog naming still matters.The new proposal won't
>>>>> really solve the naming problem.
>>>>>
>>>>> On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> Hi Dan,
>>>>>>
>>>>>> Thanks for the clarification! I agree that UUIDs should not be stored
>>>>>> in the SQL definition of the view.
>>>>>>
>>>>>> Just to clarify: the proposal isn't about modifying the view
>>>>>> definition itself, but rather about how table identifiers are represented
>>>>>> in Iceberg view metadata, which naturally depends on how those identifiers
>>>>>> are resolved.
>>>>>>
>>>>>> For example, in order to store the "storage table identifier" in the
>>>>>> metadata:
>>>>>> * Should we apply the same resolution rules based on default-catalog
>>>>>> and default-namespace?
>>>>>> * And if that resolution method is considered deterministic enough,
>>>>>> would you be comfortable not storing the UUID in the storage table
>>>>>> identifier metadata?
>>>>>>
>>>>>> The same questions apply when storing child views or tables as part
>>>>>> of the view’s lineage metadata. It would be helpful to clarify the
>>>>>> resolution model and whether UUIDs should be part of that for validation or
>>>>>> drift detection.
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> I don't think we want to include the resolved table UUIDs in the
>>>>>>> view definition, but rather in the storage table state.  You can still
>>>>>>> resolve whether those drift at some point, but I don't feel like it's a
>>>>>>> good idea to capture data in the view that we may allow to drift if there
>>>>>>> isn't any requirement that they match.  This also aligns with the
>>>>>>> identifier resolution being late binding.
>>>>>>>
>>>>>>> -Dan
>>>>>>>
>>>>>>> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks Steven! So would you agree that resolution using
>>>>>>>> default-catalog and default-namespace does not provide full determinism,
>>>>>>>> and requires a supporting safety mechanism?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Walaa.
>>>>>>>>
>>>>>>>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> > If the current model is considered deterministic, do you think
>>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>>>
>>>>>>>>> I am fine with storing UUIDs for table identifiers in the view.
>>>>>>>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>>>>>>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>>>>>>>> fail the query if mismatch.
>>>>>>>>>
>>>>>>>>> The UUID change doesn't really change the table identifier
>>>>>>>>> resolution rule though. It is more of a safety protection.
>>>>>>>>>
>>>>>>>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Steven,
>>>>>>>>>>
>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>
>>>>>>>>>> > I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>>>> Iceberg view spec.
>>>>>>>>>>
>>>>>>>>>> To clarify, I don't believe the proposal is trying to solve
>>>>>>>>>> catalog naming. What it’s doing is simply this:
>>>>>>>>>>
>>>>>>>>>> * Proposing that table names inside views resolve the same way as
>>>>>>>>>> they do elsewhere (e.g., queries).
>>>>>>>>>> * Adopting a model that is already widely used and supported in
>>>>>>>>>> the existing ecosystem, which allows for:
>>>>>>>>>>     -- Renaming catalog aliases
>>>>>>>>>>     -- Swapping catalog implementations behind consistent names
>>>>>>>>>>     -- Having different default catalog names across engines that
>>>>>>>>>> still point to the same underlying tables
>>>>>>>>>>
>>>>>>>>>> These are common patterns in production data lakes. Saying
>>>>>>>>>> Iceberg views cannot operate in those environments feels unrealistic. In
>>>>>>>>>> practice, it means the spec breaks down in situations that users encounter
>>>>>>>>>> regularly.
>>>>>>>>>>
>>>>>>>>>> > The recommendation of using engines’ current catalog and
>>>>>>>>>> database can cause context-dependent resolution results.
>>>>>>>>>>
>>>>>>>>>> * As noted in the doc and earlier replies, fixing a catalog name
>>>>>>>>>> doesn’t actually guarantee determinism either. All the failure scenarios
>>>>>>>>>> above still apply even when a default-catalog is stored.
>>>>>>>>>> * The current spec also allows default-catalog to be null, in
>>>>>>>>>> which case it falls back to the view’s catalog, yet that catalog is
>>>>>>>>>> determined based on how the view is referenced in the query, which would be
>>>>>>>>>> considered non-deterministic based on the same criteria you shared.
>>>>>>>>>> * The only true form of determinism here is UUID-based
>>>>>>>>>> validation, which protects against silent drift in any resolution model.
>>>>>>>>>>
>>>>>>>>>> If the current model is considered deterministic, do you think
>>>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>>>> Or put another way: Would you be comfortable relying solely on
>>>>>>>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>>>>>>>> table, without UUID validation?
>>>>>>>>>>
>>>>>>>>>> +1 on involving other communities. I’m happy to help facilitate a
>>>>>>>>>> cross-community discussion if we aren’t able to reach a resolution here.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Walaa.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>>>>> Iceberg view spec. I am not convinced that the proposed change will make
>>>>>>>>>>> the table identifier resolution more clear and portable. The recommendation
>>>>>>>>>>> of using engines' current catalog and database can cause context dependent
>>>>>>>>>>> resolution results, which seems non-deterministic to me.
>>>>>>>>>>>
>>>>>>>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>>>>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>>>>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>>>>>>>> more inputs from the broader community on this part.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> In Spark, I believe that the USE commands sets the current
>>>>>>>>>>>> catalog and namespace.  This affects both where the view is created and how
>>>>>>>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>>>>>>>> saving the current catalog and namespace into the view
>>>>>>>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> > I think that's the lesser evil compared to Iceberg
>>>>>>>>>>>>> specifying how engines should resolve identifiers
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think this is also similar to the previous point. It is the
>>>>>>>>>>>>> other way around. Right now the spec dictates how to resolve (through
>>>>>>>>>>>>> employing a view-specific `default-catalog` field). The proposal is
>>>>>>>>>>>>> suggesting to get out of this space and let engines handle it similar
>>>>>>>>>>>>> to how they handle all identifiers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > I thought "default-catalog" could be set via the USE
>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Benny, I think this is a misconception or miscommunication.
>>>>>>>>>>>>>> The USE command has no impact on the `default-catalog` field. In fact, the
>>>>>>>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>>>>>>>> is not the case under the current spec.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > there is no SQL construct today to explicitly set
>>>>>>>>>>>>>>> default-catalog
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I generally agree with Dan about requiring consistent
>>>>>>>>>>>>>>> catalog names.  I think that's the lesser evil compared to Iceberg
>>>>>>>>>>>>>>> specifying how engines should resolve identifiers.  Another thing to
>>>>>>>>>>>>>>> consider is that identifier resolution can be very expensive at query
>>>>>>>>>>>>>>> validation time if identifiers need to be looked up from a bunch of
>>>>>>>>>>>>>>> places.  Hopefully, it should be possible to define a view in such a way
>>>>>>>>>>>>>>> that identifiers can be resolved on the first try.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>>>>>>>> materialized views.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We may be able to converge on this storage format even if
>>>>>>>>>>>>>>>> we haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>>>>>>>> identifiers in this way.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm supportive of moving forward with consensus on the
>>>>>>>>>>>>>>>> identifier storage format. That said, we may continue to run into questions
>>>>>>>>>>>>>>>> related to resolution during implementation. For example: Should the
>>>>>>>>>>>>>>>> storage table identifier follow the same default-catalog and
>>>>>>>>>>>>>>>> default-namespace resolution behavior as other table references?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We can continue having the discussion if the current view
>>>>>>>>>>>>>>>>> spec needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Rishabh
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Correction of typo: both engines seem to set
>>>>>>>>>>>>>>>>>> default-catalog to the view catalog if it is defined, or to null if the
>>>>>>>>>>>>>>>>>> view catalog is not defined.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I agree that catalog renaming is an environmental event,
>>>>>>>>>>>>>>>>>>> but it's a real one that happens frequently in practice.
>>>>>>>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate
>>>>>>>>>>>>>>>>>>> something as common as catalog renaming feels very restrictive, and could
>>>>>>>>>>>>>>>>>>> make the spec less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>>>>>>>> I’m sharing this from the perspective of a large data
>>>>>>>>>>>>>>>>>>> lake environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Further, it's worth noting that the table spec is
>>>>>>>>>>>>>>>>>>> resilient to catalog renaming, but the view spec is not. If we have an
>>>>>>>>>>>>>>>>>>> opportunity to make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>>>>>>>> Both specifications are deterministic in their
>>>>>>>>>>>>>>>>>>> definition, but one is more fragile to environmental changes than the
>>>>>>>>>>>>>>>>>>> other. Improving resilience does not sacrifice determinism. It simply makes
>>>>>>>>>>>>>>>>>>> views safer and more portable over time.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Separately, given that there is no SQL construct today
>>>>>>>>>>>>>>>>>>> to explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>>>>>>>> Today, both engines seem to set default-catalog to null
>>>>>>>>>>>>>>>>>>> if the view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>>>>>>>> What was the intended thought process behind this
>>>>>>>>>>>>>>>>>>> behavior?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > tables inside views remain reachable after a catalog
>>>>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This problem stems from the exact
>>>>>>>>>>>>>>>>>>>> environmental/configuration issue that we should not be trying to address.
>>>>>>>>>>>>>>>>>>>> I don't think we would expect references to survive a catalog rename.
>>>>>>>>>>>>>>>>>>>> That's not something covered by the spec and needs to be handled separately
>>>>>>>>>>>>>>>>>>>> as a platform-level migration specific to the affected environment.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The identifier resolution logic is clear and
>>>>>>>>>>>>>>>>>>>> deterministic.  It should not matter whether an engine resolves and encodes
>>>>>>>>>>>>>>>>>>>> the default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but
>>>>>>>>>>>>>>>>>>>> rather view behavior when you start altering the environment around it,
>>>>>>>>>>>>>>>>>>>> which isn't something we should be trying to define here.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just
>>>>>>>>>>>>>>>>>>>>> catalog naming consistency. The behavior around default-catalog itself
>>>>>>>>>>>>>>>>>>>>> introduces resolution inconsistencies even when catalog names are
>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside
>>>>>>>>>>>>>>>>>>>>> views remain reachable after a catalog rename. But if it is set to a
>>>>>>>>>>>>>>>>>>>>> non-null value, table references will break.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * default-catalog causes table references inside views
>>>>>>>>>>>>>>>>>>>>> to be early bound (i.e., bound at view creation time, especially when using
>>>>>>>>>>>>>>>>>>>>> a non-null value), while table references inside standalone queries are
>>>>>>>>>>>>>>>>>>>>> late bound (bound at query time). This creates inconsistencies when
>>>>>>>>>>>>>>>>>>>>> resolving the same table name inside and outside views, even within the
>>>>>>>>>>>>>>>>>>>>> same job.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from
>>>>>>>>>>>>>>>>>>>>> the spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to
>>>>>>>>>>>>>>>>>>>>> Iceberg with that model. Migrated Hive views will have to unfollow that
>>>>>>>>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> How would you suggest approaching the engine-level
>>>>>>>>>>>>>>>>>>>>> changes required to support the current default-catalog field?
>>>>>>>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities
>>>>>>>>>>>>>>>>>>>>> would align around having table resolution behave inconsistently between
>>>>>>>>>>>>>>>>>>>>> queries and views, or inconsistency between Iceberg and other types of
>>>>>>>>>>>>>>>>>>>>> views?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I would agree with Jan's summary of why
>>>>>>>>>>>>>>>>>>>>>> 'default-catalog' was introduced, but I think we need to step back and
>>>>>>>>>>>>>>>>>>>>>> align on what we are really attempting to support in the spec.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent
>>>>>>>>>>>>>>>>>>>>>> to address the catalog naming/configuration and would argue that we
>>>>>>>>>>>>>>>>>>>>>> shouldn't attempt to do that as part of the spec.  Inconsistently named
>>>>>>>>>>>>>>>>>>>>>> catalogs are a reality, but we should consider that a
>>>>>>>>>>>>>>>>>>>>>> configuration/environmental issue, not something to solve for in the spec.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We should support and advocate for consistency in
>>>>>>>>>>>>>>>>>>>>>> catalog naming and define the spec along those lines.  The fact is that
>>>>>>>>>>>>>>>>>>>>>> with all of the recent work that's gone into making catalogs pluggable, it
>>>>>>>>>>>>>>>>>>>>>> makes more sense to just register catalog configuration with consistent
>>>>>>>>>>>>>>>>>>>>>> names (even if you have to duplicate the configuration for supporting
>>>>>>>>>>>>>>>>>>>>>> existing readers/writers).  I think it's better to provide a path toward
>>>>>>>>>>>>>>>>>>>>>> consistency than to normalize complicated schemes to workaround the issues
>>>>>>>>>>>>>>>>>>>>>> caused by environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late
>>>>>>>>>>>>>>>>>>>>>> binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was
>>>>>>>>>>>>>>>>>>>>>>> that every query engine uses it to store its session default catalog at the
>>>>>>>>>>>>>>>>>>>>>>> time of creating the view. This way the view could be reused in another
>>>>>>>>>>>>>>>>>>>>>>> session. The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Generally we have different environments we want to
>>>>>>>>>>>>>>>>>>>>>>> support with the view spec:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> When the environment supports it, using consistent
>>>>>>>>>>>>>>>>>>>>>>> catalog names can have a great benefit for multi-catalog, multi-engine
>>>>>>>>>>>>>>>>>>>>>>> setups. With consistent catalog names, using the "default-catalog" field
>>>>>>>>>>>>>>>>>>>>>>> works without any issues.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> This can be the case when different query engines
>>>>>>>>>>>>>>>>>>>>>>> refer to the same physical catalog by different names. This often happens
>>>>>>>>>>>>>>>>>>>>>>> because different query engines use different strategies to setup the
>>>>>>>>>>>>>>>>>>>>>>> catalogs. If catalogs have inconsistent naming, using the "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>> field does not work because it is not guaranteed that the catalog name can
>>>>>>>>>>>>>>>>>>>>>>> be resolved with another engine. Using the "view catalog" as a fallback is
>>>>>>>>>>>>>>>>>>>>>>> a better solution for this use case, as it avoids catalog names altogether.
>>>>>>>>>>>>>>>>>>>>>>> It is however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> What do you think of introducing a view property
>>>>>>>>>>>>>>>>>>>>>>> that specifies if the "default-catalog" or the "view catalog" should be
>>>>>>>>>>>>>>>>>>>>>>> used? This way, you could use the "default-catalog" in environments where
>>>>>>>>>>>>>>>>>>>>>>> you can guarantee consistent naming, but you would be able to directly
>>>>>>>>>>>>>>>>>>>>>>> fallback to the "view-catalog" when you don't have consistent naming. The
>>>>>>>>>>>>>>>>>>>>>>> query engines could set the default for this view property at creation
>>>>>>>>>>>>>>>>>>>>>>> time. Spark for example could set it to automatically use the "view
>>>>>>>>>>>>>>>>>>>>>>> catalog".
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace
>>>>>>>>>>>>>>>>>>>>>>> fields from the view spec completely.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate
>>>>>>>>>>>>>>>>>>>>>>> view-level default catalogs or namespaces.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a
>>>>>>>>>>>>>>>>>>>>>>> catalog qualifier, engines should resolve it using the current engine
>>>>>>>>>>>>>>>>>>>>>>> catalog at query time.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata
>>>>>>>>>>>>>>>>>>>>>>> exactly as they appear in the view SQL text.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at
>>>>>>>>>>>>>>>>>>>>>>> creation, it should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the
>>>>>>>>>>>>>>>>>>>>>>>> confusion the fallback creates. Also just to be clear, at this point and
>>>>>>>>>>>>>>>>>>>>>>>> after clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior
>>>>>>>>>>>>>>>>>>>>>>>>> of "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I
>>>>>>>>>>>>>>>>>>>>>>>>> don't think we should design something that says first use this catalog,
>>>>>>>>>>>>>>>>>>>>>>>>> then that catalog.. or that catalog.  For example, resolving identifiers
>>>>>>>>>>>>>>>>>>>>>>>>> using default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I
>>>>>>>>>>>>>>>>>>>>>>>>> see no value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>>>>>>>> - Interoperability between different engine types
>>>>>>>>>>>>>>>>>>>>>>>>> is very hard due to dialect issues... so I think we should focus on
>>>>>>>>>>>>>>>>>>>>>>>>> supporting different clusters of the same engine type on a shared catalog.
>>>>>>>>>>>>>>>>>>>>>>>>> For example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is
>>>>>>>>>>>>>>>>>>>>>>>>> along the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable
>>>>>>>>>>>>>>>>>>>>>>>>> identifier problem through the lens of catalog resolution but maybe the
>>>>>>>>>>>>>>>>>>>>>>>>> right approach is really about templating.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers
>>>>>>>>>>>>>>>>>>>>>>>>> in a view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the
>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistencies I shared regarding many resolution methods not consistently
>>>>>>>>>>>>>>>>>>>>>>>>>> being followed in different scenarios (view vs child table resolution,
>>>>>>>>>>>>>>>>>>>>>>>>>> query vs view resolution)? Note these occur when the default catalog is set
>>>>>>>>>>>>>>>>>>>>>>>>>> to a non-null value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>> should be the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really
>>>>>>>>>>>>>>>>>>>>>>>>>>> matter what the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog, this is a
>>>>>>>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog, catalog names
>>>>>>>>>>>>>>>>>>>>>>>>>>> should be included in the reference name
>>>>>>>>>>>>>>>>>>>>>>>>>>> already. So no ambiguity
>>>>>>>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a
>>>>>>>>>>>>>>>>>>>>>>>>>>> separate problem, which
>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can
>>>>>>>>>>>>>>>>>>>>>>>>>>> only recommend that
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across
>>>>>>>>>>>>>>>>>>>>>>>>>>> usage for better
>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior
>>>>>>>>>>>>>>>>>>>>>>>>>>> to engine's session
>>>>>>>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than
>>>>>>>>>>>>>>>>>>>>>>>>>>> the current fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation
>>>>>>>>>>>>>>>>>>>>>>>>>>> issue for not conforming
>>>>>>>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I
>>>>>>>>>>>>>>>>>>>>>>>>>>> want to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined
>>>>>>>>>>>>>>>>>>>>>>>>>>> at view creation time. It would be circular: the view needs to exist before
>>>>>>>>>>>>>>>>>>>>>>>>>>> its metadata (and hence default-catalog) can exist. This is visible in
>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>> setting would require extending SQL syntax and engine APIs to promote it to
>>>>>>>>>>>>>>>>>>>>>>>>>>> a first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs
>>>>>>>>>>>>>>>>>>>>>>>>>>> from this idea. Spark resolves table identifiers during view creation using
>>>>>>>>>>>>>>>>>>>>>>>>>>> the session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of
>>>>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" where the view is going to be created in. It can use
>>>>>>>>>>>>>>>>>>>>>>>>>>> that catalog to resolve partial table identifiers if "default-catalog" is
>>>>>>>>>>>>>>>>>>>>>>>>>>> not set.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior,
>>>>>>>>>>>>>>>>>>>>>>>>>>> where partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the
>>>>>>>>>>>>>>>>>>>>>>>>>>> tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>> How do we validate that a table exists without a default catalog, or do we
>>>>>>>>>>>>>>>>>>>>>>>>>>> always use the current session catalog?
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same
>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. Just to clarify: when I referred to late binding as
>>>>>>>>>>>>>>>>>>>>>>>>>>> “similar” to the proposal, I was acknowledging the distinction between
>>>>>>>>>>>>>>>>>>>>>>>>>>> view-level and table-level resolution. But as you noted, both follow a late
>>>>>>>>>>>>>>>>>>>>>>>>>>> binding model.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The
>>>>>>>>>>>>>>>>>>>>>>>>>>> view catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to
>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish between the physical catalog and it's reference used in SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>> statements. The important part is that the physical catalog of the view and
>>>>>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in it's definition stay consistent. You could create
>>>>>>>>>>>>>>>>>>>>>>>>>>> a view in a given physical catalog by referring to it as "catalogA", as in
>>>>>>>>>>>>>>>>>>>>>>>>>>> your first point. If you then, given a different setup, refer to the same
>>>>>>>>>>>>>>>>>>>>>>>>>>> physical catalog as "catalogB" in another session/environment, the behavior
>>>>>>>>>>>>>>>>>>>>>>>>>>> should still work.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Late binding applies to the view catalog name and by extension to all
>>>>>>>>>>>>>>>>>>>>>>>>>>> partial table references when no "default-catalog" is present. Resolving
>>>>>>>>>>>>>>>>>>>>>>>>>>> the view catalog name at query time is not opposed to storing the view
>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata in a catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what
>>>>>>>>>>>>>>>>>>>>>>>>>>> you mean.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being
>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced in a SQL statement. That statement contains the information for
>>>>>>>>>>>>>>>>>>>>>>>>>>> the query engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as
>>>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * FROM catalogB.namespace.view (after renaming catalogA to catalogB,
>>>>>>>>>>>>>>>>>>>>>>>>>>> and keeping everything else the same), then catalogB becomes the view’s
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it
>>>>>>>>>>>>>>>>>>>>>>>>>>> sounds to me like the catalog is resolved at query time, based on how the
>>>>>>>>>>>>>>>>>>>>>>>>>>> view is referenced, not from any stored metadata. That would imply some
>>>>>>>>>>>>>>>>>>>>>>>>>>> sort of a late binding behavior (similar to the proposal), as opposed to
>>>>>>>>>>>>>>>>>>>>>>>>>>> using some catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the
>>>>>>>>>>>>>>>>>>>>>>>>>>> current spec: The view is executed when it's being referenced in a SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>> statement. That statement contains the information for the query engine to
>>>>>>>>>>>>>>>>>>>>>>>>>>> resolve the catalog of the view. The query engine then uses that
>>>>>>>>>>>>>>>>>>>>>>>>>>> information to fetch the view metadata from the catalog. It also needs to
>>>>>>>>>>>>>>>>>>>>>>>>>>> temporarily keep track of which catalog it used to fetch the view metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>> It can then use that information to resolve the table references in the
>>>>>>>>>>>>>>>>>>>>>>>>>>> views SQL definition in case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>> can be referenced at execution time. As long as that's the case I would
>>>>>>>>>>>>>>>>>>>>>>>>>>> assume the view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable
>>>>>>>>>>>>>>>>>>>>>>>>>>> because the current specification can lead to some unintuitive behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>> For example for the following statement:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders
>>>>>>>>>>>>>>>>>>>>>>>>>>> AS SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the
>>>>>>>>>>>>>>>>>>>>>>>>>>> view spec, because it has the "closure" property. Because of the fact that
>>>>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" has to be known when executing a view, all the
>>>>>>>>>>>>>>>>>>>>>>>>>>> information required to resolve the table identifiers is contained in the
>>>>>>>>>>>>>>>>>>>>>>>>>>> view metadata (and the "view catalog"). I think that if you make the
>>>>>>>>>>>>>>>>>>>>>>>>>>> identifier resolution dependent on external parameters, it hinders
>>>>>>>>>>>>>>>>>>>>>>>>>>> portability.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key
>>>>>>>>>>>>>>>>>>>>>>>>>>> point before going deeper:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could
>>>>>>>>>>>>>>>>>>>>>>>>>>> you help clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog” stored, so that an engine knows to fall back to it if
>>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog is null?
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in
>>>>>>>>>>>>>>>>>>>>>>>>>>> the session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if
>>>>>>>>>>>>>>>>>>>>>>>>>>> we can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I
>>>>>>>>>>>>>>>>>>>>>>>>>>> understood correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-namepace"
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these
>>>>>>>>>>>>>>>>>>>>>>>>>>> points.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on
>>>>>>>>>>>>>>>>>>>>>>>>>>> the current session setup is in my opinion the definition of
>>>>>>>>>>>>>>>>>>>>>>>>>>> "non-determinism". You could be running the same query-engine and
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog-setup on different days, with different default session catalogs
>>>>>>>>>>>>>>>>>>>>>>>>>>> (which is rather common), and would be getting different results.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the
>>>>>>>>>>>>>>>>>>>>>>>>>>> view always produces the same results. The current behavior has some rough
>>>>>>>>>>>>>>>>>>>>>>>>>>> edges in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-namespace"
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes
>>>>>>>>>>>>>>>>>>>>>>>>>>> the setup more robust. Which is great. However, having UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>>> still requires us to have a portable table identifier specification. Even
>>>>>>>>>>>>>>>>>>>>>>>>>>> if we have the UUIDs of the referenced tables from the view, there simply
>>>>>>>>>>>>>>>>>>>>>>>>>>> isn't an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>> setup and suiting table identifiers to obtain the table metadata. We can
>>>>>>>>>>>>>>>>>>>>>>>>>>> use the UUIDs to verify if we loaded the correct table. But this can only
>>>>>>>>>>>>>>>>>>>>>>>>>>> be done after we used some identifier. Which means there is no way of using
>>>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current
>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for "default-catalog" because it is more deterministic in my
>>>>>>>>>>>>>>>>>>>>>>>>>>> opinion. And I think the current spec does a good job for multi-engine
>>>>>>>>>>>>>>>>>>>>>>>>>>> table identifier resolution. I see the UUID validation more of an
>>>>>>>>>>>>>>>>>>>>>>>>>>> additional hardening strategy.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie
>>>>>>>>>>>>>>>>>>>>>>>>>>> Liu <li...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I
>>>>>>>>>>>>>>>>>>>>>>>>>>> have some concerns with resolving catalog names on the fly with query
>>>>>>>>>>>>>>>>>>>>>>>>>>> engine defined catalog names. This introduces some flexibility at first
>>>>>>>>>>>>>>>>>>>>>>>>>>> glance, but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should
>>>>>>>>>>>>>>>>>>>>>>>>>>> store resolved table uuid in view metadata, as table/view renaming may
>>>>>>>>>>>>>>>>>>>>>>>>>>> introduce errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the
>>>>>>>>>>>>>>>>>>>>>>>>>>> momentum and closing out the MV spec as well. I’m hoping we can proceed to
>>>>>>>>>>>>>>>>>>>>>>>>>>> a vote next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps.
>>>>>>>>>>>>>>>>>>>>>>>>>>> The proposal outlines a strategy for handling table identifiers in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>> view metadata, with the goal of ensuring correctness, portability, and
>>>>>>>>>>>>>>>>>>>>>>>>>>> engine compatibility. It recommends resolving table identifiers at read
>>>>>>>>>>>>>>>>>>>>>>>>>>> time (late binding) rather than creation time, and introduces UUID-based
>>>>>>>>>>>>>>>>>>>>>>>>>>> validation to maintain identity guarantees across engines, or sessions. It
>>>>>>>>>>>>>>>>>>>>>>>>>>> also revises how default-catalog and default-namespace are handled
>>>>>>>>>>>>>>>>>>>>>>>>>>> (defaulting both to the session context if not explicitly set) to better
>>>>>>>>>>>>>>>>>>>>>>>>>>> align with engine behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have
>>>>>>>>>>>>>>>>>>>>>>>>>>> addressed the comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog names in the spec refer to logical catalogs—i.e., the first part of
>>>>>>>>>>>>>>>>>>>>>>>>>>> a three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun <su...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I
>>>>>>>>>>>>>>>>>>>>>>>>>>> think view portability is a very important topic for us to continue
>>>>>>>>>>>>>>>>>>>>>>>>>>> discussing as it relies on many assumptions within the data ecosystem for
>>>>>>>>>>>>>>>>>>>>>>>>>>> it to function like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how
>>>>>>>>>>>>>>>>>>>>>>>>>>> this may impact the permission questions the engines will be asking, and
>>>>>>>>>>>>>>>>>>>>>>>>>>> whether that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>> Eduard Tudenhöfner <et...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this
>>>>>>>>>>>>>>>>>>>>>>>>>>> problem. I've added a few comments to get a better understanding of how
>>>>>>>>>>>>>>>>>>>>>>>>>>> this will look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline
>>>>>>>>>>>>>>>>>>>>>>>>>>> with a few community members to better understand how the current spec is
>>>>>>>>>>>>>>>>>>>>>>>>>>> being interpreted. Those conversations served as inputs to a new proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>> on how table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>> I look forward to your feedback and working together to move this forward
>>>>>>>>>>>>>>>>>>>>>>>>>>> so we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Jan Kaul <ja...@mailbox.org.INVALID>]

I think it makes sense to distinguish between the identifiers for the 
storage table and the source tables.

*Storage table*: The table that stores the precomputed data of the 
materialized view. The storage table identifier is part of the view 
metadata and is used to access the precomputed data from the view.

*Source table*(Base table): Is a table referenced in the materialized 
view query. Its state is stored in the "refresh state"(lineage) as part 
of the storage table metadata to determine the freshness of the 
materialized view.

And we need to clarify if the current discussion is about table 
identifiers in general or if we're discussing either the storage table 
or source table identifiers. The following is my opinion on these:

1. Storage table identifier

It is functionally required to store a table identifier as part of the 
view metadata to be able to fetch the storage table metadata from the 
catalog. The catalog interface uses identifiers, so there is no 
alternative here. The question remains if the identifier should use 
optional namespace and catalog fields and what to do if these fields are 
not set. (I'm not sure if Walaa was also proposing the add an UUID field)

2. Source table identifiers

The source table identifiers can be obtained from parsing the SQL 
definition of the view, so it's not strictly required to store them as 
part of the metadata. At some point we agreed to use SQL parsing to 
obtain these identifiers, because this fits to how the query engine is 
executing the query. Moreover, this lets us use different 
representations for different query engines. If the representation 
translation tool supports some kind of mapping, we could have different 
"catalog aliases" with different engines. We agreed to use UUIDs to 
reference source tables in the refresh-state.
The discussion steven mentioned, was about adding the source table 
identifiers to the refresh-state(lineage) for readability, but not to be 
used by engines.
The current design doesn't allow multiple representations for the 
refresh-state(lineage). This makes it more difficult to solve the 
"catalog alias" problem as part of the refresh-state. To avoid this 
problem and since the table identifiers are not strictly required, I was 
advocating to not store any source table identifiers in the 
refresh-state. In my opinion, having one representation of source table 
identifiers in the refresh-state is in conflict with having multiple 
representations for the SQL query, where the identifiers occur.

My current questions are:

1. Are we talking about table identifiers in general or are we talking 
only about the storage table identifier or the source table identifiers?

2. When using optional fields for namespace and catalog, what are we 
using when these fields are not set? (Early vs late binding)

Thanks,

Jan

On 5/12/25 18:23, Steven Wu wrote:
>
> Regarding the following representation for the storage table identifier:
>
> * catalog name: optional
> * namespace: optional
> * table name: required
> * UUID: required
>
> It is the same discussion as this github thread.
> https://github.com/apache/iceberg/pull/11041/files#r1869998317
>
> I was in favor of adding them mostly from human readability when 
> manually inspecting the storage table to avoid a reverse lookup from 
> table UUID to table identifier. Benny (and a few others) have 
> expressed that it may add more ambiguity (like this thread was 
> discussing). To clarify, we only need to include the identifier as 
> they are defined in the SQL query (e.g. catalog and namespace are 
> optional).  I doubt it would add more ambiguity. But they are 
> certainly not functionally required. So I am not opinionated on this one.
>
> On Mon, May 12, 2025 at 5:01 AM Jan Kaul <ja...@mailbox.org.invalid> 
> wrote:
>
>     I have a question regarding the following representation for the
>     storage table identifier:
>
>     * catalog name: optional
>     * namespace: optional
>     * table name: required
>     * UUID: required
>
>     Having optional fields as part of the identifier again raises the
>     question of what to use if these optional fields are not set. And
>     we again have the early vs late binding discussion. Where early
>     binding would default to the view catalog and late binding would
>     use the current session catalog/namespace.
>
>     @Walaa: Which behavior did you have in mind?
>
>     In my opinion, having the functionality of MVs depend on the
>     current session context is not desirable. So I would argue against
>     late binding.
>
>     Best wishes,
>
>     Jan
>
>     On 5/8/25 19:59, Walaa Eldin Moustafa wrote:
>>     Let us continue here for the context and continuity of the
>>     conversation.
>>
>>     On Thu, May 8, 2025 at 10:55 AM Steven Wu <st...@gmail.com>
>>     wrote:
>>
>>         For the storage table identifier discussion, can we move to
>>         Jan's PR on materialized view spec?
>>         https://github.com/apache/iceberg/pull/11041
>>
>>
>>         On Thu, May 8, 2025 at 10:40 AM Walaa Eldin Moustafa
>>         <wa...@gmail.com> wrote:
>>
>>             It looks like we may not reach alignment on the
>>             resolution method in the short term. That said, perhaps
>>             we can move forward by aligning on the format used to
>>             store table identifiers in metadata instead.
>>
>>             The idea is that the same format could work under either
>>             resolution model, which would allow us to make progress
>>             independently of that debate.
>>
>>             Is this storage format of the storage table identifier
>>             agreeable to you?
>>             * catalog name: optional
>>             * namespace: optional
>>             * table name: required
>>             * UUID: required
>>
>>             With the condition that the components included in a
>>             table identifier in the metadata should match those used
>>             in the view text for that table.
>>
>>             The same structure above could apply to storing child
>>             tables or view identifiers as part of lineage, if and
>>             when that’s needed.
>>
>>             For the longer term resolution model, it might make sense
>>             to initiate a cross-community discussion.
>>
>>             Thanks,
>>             Walaa.
>>
>>             On Thu, May 8, 2025 at 9:07 AM Steven Wu
>>             <st...@gmail.com> wrote:
>>
>>                 If a table referenced by the view SQL query is
>>                 deleted and recreated, is the regular view still
>>                 valid? If yes, it won't be correct to use resolved
>>                 UUID to validate the name reference. On the
>>                 materialized view side, we are aligned that UUID is
>>                 necessary to detect this scenario and mark the
>>                 refresh state "stale" if a referenced table is
>>                 re-created.
>>
>>                 > esolution using default-catalog and
>>                 default-namespace does not provide full determinism,
>>
>>                 With the new proposal, the catalog and namespace
>>                 depends on the engine's current context. if the user
>>                 changes the current context (catalog and database)
>>                 via SET/USE , the view SQL query may resolve to a
>>                 different table. That is probably a bigger problem.
>>
>>                 With the current view spec, the intention is set in
>>                 the view creation time. I would rather have the view
>>                 producer set the explicit intention (than the
>>                 implicit dependency on the engine context at view
>>                 consumption phase).
>>
>>                 Also the view SQL can still reference a table with a
>>                 catalog name component. Hence the catalog naming
>>                 still matters.The new proposal won't really solve the
>>                 naming problem.
>>
>>                 On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa
>>                 <wa...@gmail.com> wrote:
>>
>>                     Hi Dan,
>>
>>                     Thanks for the clarification! I agree that UUIDs
>>                     should not be stored in the SQL definition of the
>>                     view.
>>
>>                     Just to clarify: the proposal isn't about
>>                     modifying the view definition itself, but rather
>>                     about how table identifiers are represented in
>>                     Iceberg view metadata, which naturally depends on
>>                     how those identifiers are resolved.
>>
>>                     For example, in order to store the "storage table
>>                     identifier" in the metadata:
>>                     * Should we apply the same resolution rules based
>>                     on default-catalog and default-namespace?
>>                     * And if that resolution method is considered
>>                     deterministic enough, would you be comfortable
>>                     not storing the UUID in the storage table
>>                     identifier metadata?
>>
>>                     The same questions apply when storing child views
>>                     or tables as part of the view’s lineage metadata.
>>                     It would be helpful to clarify the resolution
>>                     model and whether UUIDs should be part of that
>>                     for validation or drift detection.
>>
>>                     Thanks,
>>                     Walaa.
>>
>>
>>
>>                     On Thu, May 8, 2025 at 8:29 AM Daniel Weeks
>>                     <dw...@apache.org> wrote:
>>
>>                         I don't think we want to include the resolved
>>                         table UUIDs in the view definition, but
>>                         rather in the storage table state.  You can
>>                         still resolve whether those drift at some
>>                         point, but I don't feel like it's a good idea
>>                         to capture data in the view that we may allow
>>                         to drift if there isn't any requirement that
>>                         they match.  This also aligns with the
>>                         identifier resolution being late binding.
>>
>>                         -Dan
>>
>>                         On Wed, May 7, 2025 at 10:45 PM Walaa Eldin
>>                         Moustafa <wa...@gmail.com> wrote:
>>
>>                             Thanks Steven! So would you agree that
>>                             resolution using default-catalog and
>>                             default-namespace does not provide full
>>                             determinism, and requires a supporting
>>                             safety mechanism?
>>
>>                             Thanks,
>>                             Walaa.
>>
>>                             On Wed, May 7, 2025 at 10:30 PM Steven Wu
>>                             <st...@gmail.com> wrote:
>>
>>                                 > If the current model is considered
>>                                 deterministic, do you think
>>                                 `default-catalog` and
>>                                 `default-namespace` fields provide
>>                                 enough determinism to eliminate the
>>                                 need for UUIDs when storing table
>>                                 identifiers?
>>
>>                                 I am fine with storing UUIDs for
>>                                 table identifiers in the view.
>>                                 Basically, view creation resolves all
>>                                 referenced tables/views with UUIDs.
>>                                 View consumers can validate resolved
>>                                 tables/views with the stored UUIDs
>>                                 and fail the query if mismatch.
>>
>>                                 The UUID change doesn't really change
>>                                 the table identifier resolution rule
>>                                 though. It is more of a safety
>>                                 protection.
>>
>>                                 On Wed, May 7, 2025 at 10:02 PM Walaa
>>                                 Eldin Moustafa
>>                                 <wa...@gmail.com> wrote:
>>
>>                                     Hi Steven,
>>
>>                                     Thanks for the reply.
>>
>>                                     > I agree with Dan that we
>>                                     shouldn't solve catalog naming in
>>                                     the Iceberg view spec.
>>
>>                                     To clarify, I don't believe the
>>                                     proposal is trying to solve
>>                                     catalog naming. What it’s doing
>>                                     is simply this:
>>
>>                                     * Proposing that table names
>>                                     inside views resolve the same way
>>                                     as they do elsewhere (e.g., queries).
>>                                     * Adopting a model that is
>>                                     already widely used and supported
>>                                     in the existing ecosystem, which
>>                                     allows for:
>>                                         -- Renaming catalog aliases
>>                                         -- Swapping catalog
>>                                     implementations behind consistent
>>                                     names
>>                                         -- Having different default
>>                                     catalog names across engines that
>>                                     still point to the same
>>                                     underlying tables
>>
>>                                     These are common patterns in
>>                                     production data lakes. Saying
>>                                     Iceberg views cannot operate in
>>                                     those environments feels
>>                                     unrealistic. In practice, it
>>                                     means the spec breaks down in
>>                                     situations that users encounter
>>                                     regularly.
>>
>>                                     > The recommendation of using
>>                                     engines’ current catalog and
>>                                     database can cause
>>                                     context-dependent resolution results.
>>
>>                                     * As noted in the doc and earlier
>>                                     replies, fixing a catalog name
>>                                     doesn’t actually guarantee
>>                                     determinism either. All the
>>                                     failure scenarios above still
>>                                     apply even when a default-catalog
>>                                     is stored.
>>                                     * The current spec also allows
>>                                     default-catalog to be null, in
>>                                     which case it falls back to the
>>                                     view’s catalog, yet that catalog
>>                                     is determined based on how the
>>                                     view is referenced in the query,
>>                                     which would be considered
>>                                     non-deterministic based on the
>>                                     same criteria you shared.
>>                                     * The only true form of
>>                                     determinism here is UUID-based
>>                                     validation, which protects
>>                                     against silent drift in any
>>                                     resolution model.
>>
>>                                     If the current model is
>>                                     considered deterministic, do you
>>                                     think `default-catalog` and
>>                                     `default-namespace` fields
>>                                     provide enough determinism to
>>                                     eliminate the need for UUIDs when
>>                                     storing table identifiers?
>>                                     Or put another way: Would you be
>>                                     comfortable relying solely on
>>                                     default-catalog +
>>                                     default-namespace + table name to
>>                                     re-identify the correct table,
>>                                     without UUID validation?
>>
>>                                     +1 on involving other
>>                                     communities. I’m happy to help
>>                                     facilitate a cross-community
>>                                     discussion if we aren’t able to
>>                                     reach a resolution here.
>>
>>                                     Thanks,
>>                                     Walaa.
>>
>>
>>
>>                                     On Wed, May 7, 2025 at 9:20 PM
>>                                     Steven Wu <st...@gmail.com>
>>                                     wrote:
>>
>>                                         I agree with Dan that we
>>                                         shouldn't solve catalog
>>                                         naming in the Iceberg view
>>                                         spec. I am not convinced that
>>                                         the proposed change will make
>>                                         the table identifier
>>                                         resolution more clear and
>>                                         portable. The recommendation
>>                                         of using engines' current
>>                                         catalog and database can
>>                                         cause context dependent
>>                                         resolution results, which
>>                                         seems non-deterministic to me.
>>
>>                                         Walaa, you raised a point in
>>                                         the doc that the current
>>                                         catalog resolution logic
>>                                         (default-catalog field, then
>>                                         view catalog) is challenging
>>                                         and unrealistic for engines
>>                                         (like Spark and Trino). It
>>                                         will be great to get more
>>                                         inputs from the broader
>>                                         community on this part.
>>
>>
>>                                         On Tue, May 6, 2025 at
>>                                         9:21 AM Benny Chow
>>                                         <bt...@gmail.com> wrote:
>>
>>                                             In Spark, I believe that
>>                                             the USE commands sets the
>>                                             current catalog and
>>                                             namespace. This affects
>>                                             both where the view is
>>                                             created and how
>>                                             unqualified table
>>                                             identifiers are resolved.
>>                                             I also don't see an issue
>>                                             with saving the current
>>                                             catalog and namespace
>>                                             into the view
>>                                             metadata's default-catalog
>>                                             and default-namespace fields.
>>
>>                                             On Wed, Apr 30, 2025 at
>>                                             5:12 PM Walaa Eldin
>>                                             Moustafa
>>                                             <wa...@gmail.com>
>>                                             wrote:
>>
>>                                                 > I think that's the
>>                                                 lesser evil compared
>>                                                 to Iceberg specifying
>>                                                 how engines should
>>                                                 resolve identifiers
>>
>>                                                 I think this is also
>>                                                 similar to the
>>                                                 previous point. It is
>>                                                 the other way around.
>>                                                 Right now the spec
>>                                                 dictates how to
>>                                                 resolve (through
>>                                                 employing a
>>                                                 view-specific
>>                                                 `default-catalog`
>>                                                 field). The proposal
>>                                                 is suggesting to get
>>                                                 out of this space and
>>                                                 let engines handle it
>>                                                 similar to how
>>                                                 they handle all
>>                                                 identifiers.
>>
>>                                                 On Wed, Apr 30, 2025
>>                                                 at 5:07 PM Walaa
>>                                                 Eldin Moustafa
>>                                                 <wa...@gmail.com>
>>                                                 wrote:
>>
>>                                                     > I thought
>>                                                     "default-catalog"
>>                                                     could be set via
>>                                                     the USE command.
>>
>>                                                     Benny, I think
>>                                                     this is a
>>                                                     misconception or
>>                                                     miscommunication.
>>                                                     The USE command
>>                                                     has no impact on
>>                                                     the
>>                                                     `default-catalog`
>>                                                     field. In fact,
>>                                                     the proposal's
>>                                                     direction is
>>                                                     exactly to
>>                                                     establish that
>>                                                     USE command
>>                                                     should influence
>>                                                     how tables are
>>                                                     resolved, same
>>                                                     like everywhere
>>                                                     else. Right now
>>                                                     it is not the
>>                                                     case under the
>>                                                     current spec.
>>
>>
>>                                                     On Wed, Apr 30,
>>                                                     2025 at 3:17 PM
>>                                                     Benny Chow
>>                                                     <bt...@gmail.com>
>>                                                     wrote:
>>
>>                                                         > there is no
>>                                                         SQL construct
>>                                                         today to
>>                                                         explicitly
>>                                                         set
>>                                                         default-catalog
>>
>>                                                         I thought
>>                                                         "default-catalog"
>>                                                         could be set
>>                                                         via the USE
>>                                                         command.
>>
>>                                                         I generally
>>                                                         agree with
>>                                                         Dan about
>>                                                         requiring
>>                                                         consistent
>>                                                         catalog
>>                                                         names.  I
>>                                                         think that's
>>                                                         the lesser
>>                                                         evil compared
>>                                                         to Iceberg
>>                                                         specifying
>>                                                         how engines
>>                                                         should
>>                                                         resolve
>>                                                         identifiers.
>>                                                         Another thing
>>                                                         to consider
>>                                                         is that
>>                                                         identifier
>>                                                         resolution
>>                                                         can be very
>>                                                         expensive at
>>                                                         query
>>                                                         validation
>>                                                         time if
>>                                                         identifiers
>>                                                         need to be
>>                                                         looked up
>>                                                         from a bunch
>>                                                         of places.
>>                                                         Hopefully, it
>>                                                         should be
>>                                                         possible to
>>                                                         define a view
>>                                                         in such a way
>>                                                         that
>>                                                         identifiers
>>                                                         can be
>>                                                         resolved on
>>                                                         the first try.
>>
>>                                                         Benny
>>
>>                                                         On Tue, Apr
>>                                                         29, 2025 at
>>                                                         10:29 PM
>>                                                         Walaa Eldin
>>                                                         Moustafa
>>                                                         <wa...@gmail.com>
>>                                                         wrote:
>>
>>                                                             Hi Rishabh,
>>
>>                                                             You're
>>                                                             right
>>                                                             that the
>>                                                             proposal
>>                                                             touches
>>                                                             on two
>>                                                             aspects,
>>                                                             and
>>                                                             resolution
>>                                                             rules are
>>                                                             one of
>>                                                             them. The
>>                                                             other
>>                                                             aspect is
>>                                                             the
>>                                                             proposal's
>>                                                             position
>>                                                             that
>>                                                             table
>>                                                             identifiers
>>                                                             should be
>>                                                             stored in
>>                                                             metadata
>>                                                             exactly
>>                                                             as they
>>                                                             appear in
>>                                                             the view
>>                                                             text
>>                                                             (e.g.,
>>                                                             even if
>>                                                             they're
>>                                                             two-part
>>                                                             or
>>                                                             partially
>>                                                             qualified),
>>                                                             along
>>                                                             with
>>                                                             their
>>                                                             corresponding
>>                                                             UUIDs for
>>                                                             validation.
>>                                                             This
>>                                                             applies
>>                                                             both to
>>                                                             referenced
>>                                                             input
>>                                                             tables
>>                                                             and the
>>                                                             storage
>>                                                             table
>>                                                             identifier
>>                                                             in
>>                                                             materialized
>>                                                             views.
>>
>>                                                             We may be
>>                                                             able to
>>                                                             converge
>>                                                             on this
>>                                                             storage
>>                                                             format
>>                                                             even if
>>                                                             we
>>                                                             haven't
>>                                                             yet
>>                                                             converged
>>                                                             on the
>>                                                             resolution
>>                                                             fallback
>>                                                             rules. I
>>                                                             believe
>>                                                             both
>>                                                             resolution
>>                                                             strategies
>>                                                             currently
>>                                                             being
>>                                                             discussed
>>                                                             would
>>                                                             still
>>                                                             lead to
>>                                                             storing
>>                                                             identifiers
>>                                                             in this way.
>>
>>                                                             I'm
>>                                                             supportive
>>                                                             of moving
>>                                                             forward
>>                                                             with
>>                                                             consensus
>>                                                             on the
>>                                                             identifier
>>                                                             storage
>>                                                             format.
>>                                                             That
>>                                                             said, we
>>                                                             may
>>                                                             continue
>>                                                             to run
>>                                                             into
>>                                                             questions
>>                                                             related
>>                                                             to
>>                                                             resolution
>>                                                             during
>>                                                             implementation.
>>                                                             For
>>                                                             example:
>>                                                             Should
>>                                                             the
>>                                                             storage
>>                                                             table
>>                                                             identifier
>>                                                             follow
>>                                                             the same
>>                                                             default-catalog
>>                                                             and
>>                                                             default-namespace
>>                                                             resolution
>>                                                             behavior
>>                                                             as other
>>                                                             table
>>                                                             references?
>>
>>                                                             Thanks,
>>                                                             Walaa.
>>
>>                                                             On Tue,
>>                                                             Apr 29,
>>                                                             2025 at
>>                                                             10:07 PM
>>                                                             Rishabh
>>                                                             Bhatia
>>                                                             <bh...@gmail.com>
>>                                                             wrote:
>>
>>                                                                 Hello
>>                                                                 Walaa,
>>
>>                                                                 Thanks
>>                                                                 for
>>                                                                 starting
>>                                                                 this
>>                                                                 discussion.
>>
>>
>>                                                                 I
>>                                                                 think
>>                                                                 we
>>                                                                 should
>>                                                                 decouple
>>                                                                 at
>>                                                                 least
>>                                                                 the
>>                                                                 MV
>>                                                                 Spec
>>                                                                 from
>>                                                                 the
>>                                                                 proposal
>>                                                                 to
>>                                                                 change
>>                                                                 the
>>                                                                 current
>>                                                                 behavior
>>                                                                 of
>>                                                                 view
>>                                                                 resolution.
>>
>>
>>                                                                 We
>>                                                                 can
>>                                                                 continue
>>                                                                 having
>>                                                                 the
>>                                                                 discussion
>>                                                                 if
>>                                                                 the
>>                                                                 current
>>                                                                 view
>>                                                                 spec
>>                                                                 needs
>>                                                                 to be
>>                                                                 changed
>>                                                                 or
>>                                                                 not.
>>                                                                 Based
>>                                                                 on
>>                                                                 the
>>                                                                 decision
>>                                                                 at a
>>                                                                 later
>>                                                                 point
>>                                                                 if
>>                                                                 required
>>                                                                 we
>>                                                                 can
>>                                                                 update
>>                                                                 the
>>                                                                 view
>>                                                                 resolution
>>                                                                 rule.
>>
>>
>>                                                                 Thanks,
>>                                                                 Rishabh
>>
>>                                                                 On
>>                                                                 Mon,
>>                                                                 Apr
>>                                                                 28,
>>                                                                 2025
>>                                                                 at
>>                                                                 3:22 PM
>>                                                                 Walaa
>>                                                                 Eldin
>>                                                                 Moustafa
>>                                                                 <wa...@gmail.com>
>>                                                                 wrote:
>>
>>                                                                     Correction
>>                                                                     of
>>                                                                     typo:
>>                                                                     both
>>                                                                     engines
>>                                                                     seem
>>                                                                     to
>>                                                                     set
>>                                                                     default-catalog
>>                                                                     to
>>                                                                     the
>>                                                                     view
>>                                                                     catalog
>>                                                                     if
>>                                                                     it
>>                                                                     is
>>                                                                     defined,
>>                                                                     or
>>                                                                     to
>>                                                                     null
>>                                                                     if
>>                                                                     the
>>                                                                     view
>>                                                                     catalog
>>                                                                     is
>>                                                                     not
>>                                                                     defined.
>>
>>                                                                     On
>>                                                                     Mon,
>>                                                                     Apr
>>                                                                     28,
>>                                                                     2025
>>                                                                     at
>>                                                                     3:06 PM
>>                                                                     Walaa
>>                                                                     Eldin
>>                                                                     Moustafa
>>                                                                     <wa...@gmail.com>
>>                                                                     wrote:
>>
>>                                                                         Hi
>>                                                                         Dan,
>>
>>                                                                         Thanks
>>                                                                         again
>>                                                                         for
>>                                                                         your
>>                                                                         response.
>>
>>                                                                         I
>>                                                                         agree
>>                                                                         that
>>                                                                         catalog
>>                                                                         renaming
>>                                                                         is
>>                                                                         an
>>                                                                         environmental
>>                                                                         event,
>>                                                                         but
>>                                                                         it's
>>                                                                         a
>>                                                                         real
>>                                                                         one
>>                                                                         that
>>                                                                         happens
>>                                                                         frequently
>>                                                                         in
>>                                                                         practice.
>>                                                                         Saying
>>                                                                         that
>>                                                                         the
>>                                                                         Iceberg
>>                                                                         spec
>>                                                                         cannot
>>                                                                         accommodate
>>                                                                         something
>>                                                                         as
>>                                                                         common
>>                                                                         as
>>                                                                         catalog
>>                                                                         renaming
>>                                                                         feels
>>                                                                         very
>>                                                                         restrictive,
>>                                                                         and
>>                                                                         could
>>                                                                         make
>>                                                                         the
>>                                                                         spec
>>                                                                         less
>>                                                                         practical,
>>                                                                         even
>>                                                                         unusable,
>>                                                                         for
>>                                                                         real-world
>>                                                                         deployments.
>>                                                                         I’m
>>                                                                         sharing
>>                                                                         this
>>                                                                         from
>>                                                                         the
>>                                                                         perspective
>>                                                                         of
>>                                                                         a
>>                                                                         large
>>                                                                         data
>>                                                                         lake
>>                                                                         environment
>>                                                                         where
>>                                                                         views
>>                                                                         are
>>                                                                         heavily
>>                                                                         deployed
>>                                                                         and
>>                                                                         operationalized.
>>
>>                                                                         Further,
>>                                                                         it's
>>                                                                         worth
>>                                                                         noting
>>                                                                         that
>>                                                                         the
>>                                                                         table
>>                                                                         spec
>>                                                                         is
>>                                                                         resilient
>>                                                                         to
>>                                                                         catalog
>>                                                                         renaming,
>>                                                                         but
>>                                                                         the
>>                                                                         view
>>                                                                         spec
>>                                                                         is
>>                                                                         not.
>>                                                                         If
>>                                                                         we
>>                                                                         have
>>                                                                         an
>>                                                                         opportunity
>>                                                                         to
>>                                                                         make
>>                                                                         the
>>                                                                         view
>>                                                                         spec
>>                                                                         similarly
>>                                                                         resilient,
>>                                                                         I
>>                                                                         wonder
>>                                                                         why
>>                                                                         not?
>>                                                                         Both
>>                                                                         specifications
>>                                                                         are
>>                                                                         deterministic
>>                                                                         in
>>                                                                         their
>>                                                                         definition,
>>                                                                         but
>>                                                                         one
>>                                                                         is
>>                                                                         more
>>                                                                         fragile
>>                                                                         to
>>                                                                         environmental
>>                                                                         changes
>>                                                                         than
>>                                                                         the
>>                                                                         other.
>>                                                                         Improving
>>                                                                         resilience
>>                                                                         does
>>                                                                         not
>>                                                                         sacrifice
>>                                                                         determinism.
>>                                                                         It
>>                                                                         simply
>>                                                                         makes
>>                                                                         views
>>                                                                         safer
>>                                                                         and
>>                                                                         more
>>                                                                         portable
>>                                                                         over
>>                                                                         time.
>>
>>                                                                         Separately,
>>                                                                         given
>>                                                                         that
>>                                                                         there
>>                                                                         is
>>                                                                         no
>>                                                                         SQL
>>                                                                         construct
>>                                                                         today
>>                                                                         to
>>                                                                         explicitly
>>                                                                         set
>>                                                                         default-catalog
>>                                                                         at
>>                                                                         creation
>>                                                                         time, what
>>                                                                         is
>>                                                                         the
>>                                                                         intuition
>>                                                                         behind
>>                                                                         how
>>                                                                         engines
>>                                                                         like
>>                                                                         Spark
>>                                                                         and
>>                                                                         Trino
>>                                                                         currently
>>                                                                         assign
>>                                                                         default-catalog?
>>                                                                         Today,
>>                                                                         both
>>                                                                         engines
>>                                                                         seem
>>                                                                         to
>>                                                                         set
>>                                                                         default-catalog
>>                                                                         to
>>                                                                         null
>>                                                                         if
>>                                                                         the
>>                                                                         view
>>                                                                         catalog
>>                                                                         is
>>                                                                         defined,
>>                                                                         or
>>                                                                         to
>>                                                                         the
>>                                                                         view
>>                                                                         catalog
>>                                                                         if
>>                                                                         not.
>>                                                                         What
>>                                                                         was
>>                                                                         the
>>                                                                         intended
>>                                                                         thought
>>                                                                         process
>>                                                                         behind
>>                                                                         this
>>                                                                         behavior?
>>
>>                                                                         Thanks,
>>                                                                         Walaa
>>
>>                                                                         On
>>                                                                         Mon,
>>                                                                         Apr
>>                                                                         28,
>>                                                                         2025
>>                                                                         at
>>                                                                         1:33 PM
>>                                                                         Daniel
>>                                                                         Weeks
>>                                                                         <dw...@apache.org>
>>                                                                         wrote:
>>
>>                                                                             Walaa,
>>
>>
>>                                                                             >
>>                                                                             tables
>>                                                                             inside
>>                                                                             views
>>                                                                             remain
>>                                                                             reachable
>>                                                                             after
>>                                                                             a
>>                                                                             catalog
>>                                                                             rename
>>
>>                                                                             This
>>                                                                             problem
>>                                                                             stems
>>                                                                             from
>>                                                                             the exact
>>                                                                             environmental/configuration
>>                                                                             issue
>>                                                                             that
>>                                                                             we
>>                                                                             should
>>                                                                             not
>>                                                                             be
>>                                                                             trying
>>                                                                             to
>>                                                                             address. 
>>                                                                             I
>>                                                                             don't
>>                                                                             think
>>                                                                             we
>>                                                                             would
>>                                                                             expect
>>                                                                             references
>>                                                                             to
>>                                                                             survive
>>                                                                             a
>>                                                                             catalog
>>                                                                             rename.
>>                                                                             That's
>>                                                                             not
>>                                                                             something
>>                                                                             covered
>>                                                                             by
>>                                                                             the
>>                                                                             spec
>>                                                                             and
>>                                                                             needs
>>                                                                             to
>>                                                                             be
>>                                                                             handled
>>                                                                             separately
>>                                                                             as
>>                                                                             a
>>                                                                             platform-level
>>                                                                             migration
>>                                                                             specific
>>                                                                             to
>>                                                                             the
>>                                                                             affected
>>                                                                             environment.
>>
>>                                                                             The
>>                                                                             identifier
>>                                                                             resolution
>>                                                                             logic
>>                                                                             is
>>                                                                             clear
>>                                                                             and
>>                                                                             deterministic.
>>                                                                             It
>>                                                                             should
>>                                                                             not
>>                                                                             matter
>>                                                                             whether
>>                                                                             an
>>                                                                             engine
>>                                                                             resolves
>>                                                                             and
>>                                                                             encodes
>>                                                                             the
>>                                                                             default-catalog
>>                                                                             or
>>                                                                             leaves
>>                                                                             it
>>                                                                             to
>>                                                                             the
>>                                                                             resolution
>>                                                                             rules.
>>
>>                                                                             The
>>                                                                             issue
>>                                                                             isn't
>>                                                                             with
>>                                                                             how
>>                                                                             the
>>                                                                             spec
>>                                                                             is
>>                                                                             defined,
>>                                                                             but
>>                                                                             rather
>>                                                                             view
>>                                                                             behavior
>>                                                                             when
>>                                                                             you
>>                                                                             start
>>                                                                             altering
>>                                                                             the
>>                                                                             environment
>>                                                                             around
>>                                                                             it,
>>                                                                             which
>>                                                                             isn't
>>                                                                             something we
>>                                                                             should
>>                                                                             be
>>                                                                             trying
>>                                                                             to
>>                                                                             define
>>                                                                             here.
>>
>>                                                                             -Dan
>>
>>                                                                             On
>>                                                                             Mon,
>>                                                                             Apr
>>                                                                             28,
>>                                                                             2025
>>                                                                             at
>>                                                                             12:17 PM
>>                                                                             Walaa
>>                                                                             Eldin
>>                                                                             Moustafa
>>                                                                             <wa...@gmail.com>
>>                                                                             wrote:
>>
>>                                                                                 Hi
>>                                                                                 Dan,
>>
>>                                                                                 Thanks
>>                                                                                 for
>>                                                                                 chiming
>>                                                                                 in.
>>
>>                                                                                 I
>>                                                                                 believe
>>                                                                                 the
>>                                                                                 issues
>>                                                                                 we’re
>>                                                                                 seeing
>>                                                                                 now
>>                                                                                 go
>>                                                                                 beyond
>>                                                                                 just
>>                                                                                 catalog
>>                                                                                 naming
>>                                                                                 consistency.
>>                                                                                 The
>>                                                                                 behavior
>>                                                                                 around
>>                                                                                 default-catalog
>>                                                                                 itself
>>                                                                                 introduces
>>                                                                                 resolution
>>                                                                                 inconsistencies
>>                                                                                 even
>>                                                                                 when
>>                                                                                 catalog
>>                                                                                 names
>>                                                                                 are
>>                                                                                 consistent.
>>                                                                                 For
>>                                                                                 example:
>>
>>                                                                                 *
>>                                                                                 When
>>                                                                                 default-catalog
>>                                                                                 is
>>                                                                                 set
>>                                                                                 to
>>                                                                                 null,
>>                                                                                 tables
>>                                                                                 inside
>>                                                                                 views
>>                                                                                 remain
>>                                                                                 reachable
>>                                                                                 after
>>                                                                                 a
>>                                                                                 catalog
>>                                                                                 rename.
>>                                                                                 But
>>                                                                                 if
>>                                                                                 it
>>                                                                                 is
>>                                                                                 set
>>                                                                                 to
>>                                                                                 a
>>                                                                                 non-null
>>                                                                                 value,
>>                                                                                 table
>>                                                                                 references
>>                                                                                 will
>>                                                                                 break.
>>
>>                                                                                 *
>>                                                                                 default-catalog
>>                                                                                 causes
>>                                                                                 table
>>                                                                                 references
>>                                                                                 inside
>>                                                                                 views
>>                                                                                 to
>>                                                                                 be
>>                                                                                 early
>>                                                                                 bound
>>                                                                                 (i.e.,
>>                                                                                 bound
>>                                                                                 at
>>                                                                                 view
>>                                                                                 creation
>>                                                                                 time,
>>                                                                                 especially
>>                                                                                 when
>>                                                                                 using
>>                                                                                 a
>>                                                                                 non-null
>>                                                                                 value),
>>                                                                                 while
>>                                                                                 table
>>                                                                                 references
>>                                                                                 inside
>>                                                                                 standalone
>>                                                                                 queries
>>                                                                                 are
>>                                                                                 late
>>                                                                                 bound
>>                                                                                 (bound
>>                                                                                 at
>>                                                                                 query
>>                                                                                 time).
>>                                                                                 This
>>                                                                                 creates
>>                                                                                 inconsistencies
>>                                                                                 when
>>                                                                                 resolving
>>                                                                                 the
>>                                                                                 same
>>                                                                                 table
>>                                                                                 name
>>                                                                                 inside
>>                                                                                 and
>>                                                                                 outside
>>                                                                                 views,
>>                                                                                 even
>>                                                                                 within
>>                                                                                 the
>>                                                                                 same
>>                                                                                 job.
>>
>>                                                                                 *
>>                                                                                 It
>>                                                                                 causes
>>                                                                                 Spark's
>>                                                                                 and
>>                                                                                 Trino
>>                                                                                 behavior
>>                                                                                 to
>>                                                                                 drift
>>                                                                                 from
>>                                                                                 the
>>                                                                                 spec.
>>                                                                                 There
>>                                                                                 is
>>                                                                                 no
>>                                                                                 way
>>                                                                                 to
>>                                                                                 fully
>>                                                                                 align
>>                                                                                 Spark's
>>                                                                                 behavior
>>                                                                                 without
>>                                                                                 making
>>                                                                                 invasive
>>                                                                                 changes
>>                                                                                 to
>>                                                                                 the
>>                                                                                 Spark
>>                                                                                 SQL
>>                                                                                 grammar
>>                                                                                 and
>>                                                                                 the
>>                                                                                 View
>>                                                                                 DataSource
>>                                                                                 API
>>                                                                                 (specifically
>>                                                                                 on
>>                                                                                 the
>>                                                                                 CREATE
>>                                                                                 side).
>>                                                                                 This
>>                                                                                 challenge
>>                                                                                 would
>>                                                                                 extend
>>                                                                                 to
>>                                                                                 other
>>                                                                                 engines
>>                                                                                 too.
>>                                                                                 Both
>>                                                                                 Spark
>>                                                                                 and
>>                                                                                 Trino
>>                                                                                 set
>>                                                                                 this
>>                                                                                 field
>>                                                                                 based
>>                                                                                 on
>>                                                                                 a
>>                                                                                 heuristic
>>                                                                                 in
>>                                                                                 today's
>>                                                                                 implementation.
>>
>>                                                                                 *
>>                                                                                 With
>>                                                                                 view
>>                                                                                 nesting
>>                                                                                 (views
>>                                                                                 depending
>>                                                                                 on
>>                                                                                 views),
>>                                                                                 these
>>                                                                                 inconsistencies
>>                                                                                 amplify
>>                                                                                 further,
>>                                                                                 forcing
>>                                                                                 users
>>                                                                                 and
>>                                                                                 engines
>>                                                                                 to
>>                                                                                 reason
>>                                                                                 about
>>                                                                                 catalog
>>                                                                                 resolution
>>                                                                                 at
>>                                                                                 every
>>                                                                                 level
>>                                                                                 in
>>                                                                                 the
>>                                                                                 view
>>                                                                                 tree.
>>
>>                                                                                 *
>>                                                                                 It
>>                                                                                 will
>>                                                                                 be
>>                                                                                 difficult
>>                                                                                 to
>>                                                                                 migrate
>>                                                                                 Hive
>>                                                                                 views
>>                                                                                 to
>>                                                                                 Iceberg
>>                                                                                 with
>>                                                                                 that
>>                                                                                 model.
>>                                                                                 Migrated
>>                                                                                 Hive
>>                                                                                 views
>>                                                                                 will
>>                                                                                 have
>>                                                                                 to
>>                                                                                 unfollow
>>                                                                                 that
>>                                                                                 spec.
>>
>>                                                                                 How
>>                                                                                 would
>>                                                                                 you
>>                                                                                 suggest
>>                                                                                 approaching
>>                                                                                 the
>>                                                                                 engine-level
>>                                                                                 changes
>>                                                                                 required
>>                                                                                 to
>>                                                                                 support
>>                                                                                 the
>>                                                                                 current
>>                                                                                 default-catalog
>>                                                                                 field?
>>                                                                                 Also,
>>                                                                                 do
>>                                                                                 you
>>                                                                                 believe
>>                                                                                 the
>>                                                                                 Spark
>>                                                                                 and
>>                                                                                 Trino
>>                                                                                 communities
>>                                                                                 would
>>                                                                                 align
>>                                                                                 around
>>                                                                                 having
>>                                                                                 table
>>                                                                                 resolution
>>                                                                                 behave
>>                                                                                 inconsistently
>>                                                                                 between
>>                                                                                 queries
>>                                                                                 and
>>                                                                                 views, or
>>                                                                                 inconsistency
>>                                                                                 between
>>                                                                                 Iceberg
>>                                                                                 and
>>                                                                                 other
>>                                                                                 types
>>                                                                                 of
>>                                                                                 views?
>>
>>                                                                                 Thanks,
>>                                                                                 Walaa
>>
>>
>>
>>                                                                                 On
>>                                                                                 Mon,
>>                                                                                 Apr
>>                                                                                 28,
>>                                                                                 2025
>>                                                                                 at
>>                                                                                 11:34 AM
>>                                                                                 Daniel
>>                                                                                 Weeks
>>                                                                                 <dw...@apache.org>
>>                                                                                 wrote:
>>
>>                                                                                     I
>>                                                                                     would
>>                                                                                     agree
>>                                                                                     with
>>                                                                                     Jan's
>>                                                                                     summary
>>                                                                                     of
>>                                                                                     why
>>                                                                                     'default-catalog'
>>                                                                                     was
>>                                                                                     introduced,
>>                                                                                     but
>>                                                                                     I
>>                                                                                     think
>>                                                                                     we
>>                                                                                     need
>>                                                                                     to
>>                                                                                     step
>>                                                                                     back
>>                                                                                     and
>>                                                                                     align
>>                                                                                     on
>>                                                                                     what
>>                                                                                     we
>>                                                                                     are
>>                                                                                     really
>>                                                                                     attempting
>>                                                                                     to
>>                                                                                     support
>>                                                                                     in
>>                                                                                     the
>>                                                                                     spec.
>>
>>
>>                                                                                     The
>>                                                                                     issues
>>                                                                                     we're
>>                                                                                     discussing
>>                                                                                     largely
>>                                                                                     stem
>>                                                                                     from
>>                                                                                     using
>>                                                                                     multiple
>>                                                                                     engines
>>                                                                                     with
>>                                                                                     cross
>>                                                                                     catalog
>>                                                                                     references
>>                                                                                     and
>>                                                                                     configurations
>>                                                                                     where
>>                                                                                     catalog
>>                                                                                     names
>>                                                                                     are
>>                                                                                     not
>>                                                                                     aligned. 
>>                                                                                     If
>>                                                                                     we
>>                                                                                     have
>>                                                                                     multiple
>>                                                                                     engines
>>                                                                                     that
>>                                                                                     all
>>                                                                                     have
>>                                                                                     the
>>                                                                                     same
>>                                                                                     catalog
>>                                                                                     names/configurations,
>>                                                                                     the
>>                                                                                     current
>>                                                                                     spec
>>                                                                                     implementation
>>                                                                                     is
>>                                                                                     well
>>                                                                                     defined
>>                                                                                     for
>>                                                                                     table
>>                                                                                     resolution
>>                                                                                     even
>>                                                                                     across
>>                                                                                     catalogs. 
>>                                                                                     The
>>                                                                                     'default-catalog'
>>                                                                                     (and
>>                                                                                     namespace
>>                                                                                     equivalent)
>>                                                                                     was
>>                                                                                     intended
>>                                                                                     to
>>                                                                                     address
>>                                                                                     the
>>                                                                                     resolution
>>                                                                                     within
>>                                                                                     the
>>                                                                                     context
>>                                                                                     of
>>                                                                                     the
>>                                                                                     sql
>>                                                                                     text,
>>                                                                                     not
>>                                                                                     to
>>                                                                                     address
>>                                                                                     catalog/naming
>>                                                                                     inconsistencies.
>>
>>                                                                                     I
>>                                                                                     feel
>>                                                                                     like
>>                                                                                     we're
>>                                                                                     trying
>>                                                                                     to
>>                                                                                     adapt the
>>                                                                                     original
>>                                                                                     intent
>>                                                                                     to
>>                                                                                     address
>>                                                                                     the
>>                                                                                     catalog
>>                                                                                     naming/configuration
>>                                                                                     and
>>                                                                                     would
>>                                                                                     argue
>>                                                                                     that
>>                                                                                     we
>>                                                                                     shouldn't
>>                                                                                     attempt
>>                                                                                     to
>>                                                                                     do
>>                                                                                     that
>>                                                                                     as
>>                                                                                     part
>>                                                                                     of
>>                                                                                     the
>>                                                                                     spec.
>>                                                                                     Inconsistently
>>                                                                                     named
>>                                                                                     catalogs
>>                                                                                     are
>>                                                                                     a
>>                                                                                     reality,
>>                                                                                     but
>>                                                                                     we
>>                                                                                     should consider
>>                                                                                     that
>>                                                                                     a
>>                                                                                     configuration/environmental
>>                                                                                     issue,
>>                                                                                     not
>>                                                                                     something
>>                                                                                     to
>>                                                                                     solve
>>                                                                                     for
>>                                                                                     in
>>                                                                                     the
>>                                                                                     spec.
>>
>>
>>                                                                                     We
>>                                                                                     should
>>                                                                                     support
>>                                                                                     and
>>                                                                                     advocate
>>                                                                                     for
>>                                                                                     consistency
>>                                                                                     in
>>                                                                                     catalog
>>                                                                                     naming
>>                                                                                     and
>>                                                                                     define
>>                                                                                     the
>>                                                                                     spec
>>                                                                                     along
>>                                                                                     those
>>                                                                                     lines.
>>                                                                                     The
>>                                                                                     fact
>>                                                                                     is
>>                                                                                     that
>>                                                                                     with
>>                                                                                     all
>>                                                                                     of
>>                                                                                     the
>>                                                                                     recent
>>                                                                                     work
>>                                                                                     that's
>>                                                                                     gone
>>                                                                                     into
>>                                                                                     making
>>                                                                                     catalogs
>>                                                                                     pluggable,
>>                                                                                     it
>>                                                                                     makes
>>                                                                                     more
>>                                                                                     sense
>>                                                                                     to
>>                                                                                     just
>>                                                                                     register
>>                                                                                     catalog
>>                                                                                     configuration
>>                                                                                     with
>>                                                                                     consistent
>>                                                                                     names
>>                                                                                     (even
>>                                                                                     if
>>                                                                                     you
>>                                                                                     have
>>                                                                                     to
>>                                                                                     duplicate
>>                                                                                     the
>>                                                                                     configuration
>>                                                                                     for
>>                                                                                     supporting
>>                                                                                     existing
>>                                                                                     readers/writers).
>>                                                                                     I
>>                                                                                     think
>>                                                                                     it's
>>                                                                                     better
>>                                                                                     to
>>                                                                                     provide
>>                                                                                     a
>>                                                                                     path
>>                                                                                     toward
>>                                                                                     consistency
>>                                                                                     than
>>                                                                                     to
>>                                                                                     normalize
>>                                                                                     complicated
>>                                                                                     schemes
>>                                                                                     to
>>                                                                                     workaround
>>                                                                                     the
>>                                                                                     issues
>>                                                                                     caused
>>                                                                                     by
>>                                                                                     environmental/configuration
>>                                                                                     inconsistencies.
>>
>>                                                                                     If
>>                                                                                     the
>>                                                                                     goal
>>                                                                                     is
>>                                                                                     to
>>                                                                                     create
>>                                                                                     clever
>>                                                                                     ways
>>                                                                                     to
>>                                                                                     hack
>>                                                                                     the
>>                                                                                     late
>>                                                                                     binding
>>                                                                                     resolution
>>                                                                                     to
>>                                                                                     swap
>>                                                                                     in
>>                                                                                     different
>>                                                                                     catalogs
>>                                                                                     or
>>                                                                                     make
>>                                                                                     references
>>                                                                                     contextual,
>>                                                                                     I
>>                                                                                     feel
>>                                                                                     like
>>                                                                                     that
>>                                                                                     is
>>                                                                                     something
>>                                                                                     we
>>                                                                                     should
>>                                                                                     strongly
>>                                                                                     discourage
>>                                                                                     as
>>                                                                                     it
>>                                                                                     leads
>>                                                                                     to
>>                                                                                     confusion
>>                                                                                     about
>>                                                                                     what
>>                                                                                     is
>>                                                                                     resolved
>>                                                                                     as
>>                                                                                     part
>>                                                                                     of
>>                                                                                     the
>>                                                                                     query.
>>
>>                                                                                     At
>>                                                                                     this
>>                                                                                     point,
>>                                                                                     I
>>                                                                                     don't
>>                                                                                     see
>>                                                                                     a
>>                                                                                     good
>>                                                                                     argument
>>                                                                                     to
>>                                                                                     add
>>                                                                                     additional configuration
>>                                                                                     or
>>                                                                                     change
>>                                                                                     the
>>                                                                                     resolution
>>                                                                                     behaviors.
>>
>>                                                                                     -Dan
>>
>>
>>
>>
>>                                                                                     On
>>                                                                                     Mon,
>>                                                                                     Apr
>>                                                                                     28,
>>                                                                                     2025
>>                                                                                     at
>>                                                                                     12:40 AM
>>                                                                                     Jan
>>                                                                                     Kaul
>>                                                                                     <ja...@mailbox.org.invalid>
>>                                                                                     <ma...@mailbox.org.invalid>
>>                                                                                     wrote:
>>
>>                                                                                         I
>>                                                                                         think
>>                                                                                         the
>>                                                                                         intention
>>                                                                                         with
>>                                                                                         the
>>                                                                                         "default-catalog"
>>                                                                                         was
>>                                                                                         that
>>                                                                                         every
>>                                                                                         query
>>                                                                                         engine
>>                                                                                         uses
>>                                                                                         it
>>                                                                                         to
>>                                                                                         store
>>                                                                                         its
>>                                                                                         session
>>                                                                                         default
>>                                                                                         catalog
>>                                                                                         at
>>                                                                                         the
>>                                                                                         time
>>                                                                                         of
>>                                                                                         creating
>>                                                                                         the
>>                                                                                         view.
>>                                                                                         This
>>                                                                                         way
>>                                                                                         the
>>                                                                                         view
>>                                                                                         could
>>                                                                                         be
>>                                                                                         reused
>>                                                                                         in
>>                                                                                         another
>>                                                                                         session.
>>                                                                                         The
>>                                                                                         idea
>>                                                                                         was
>>                                                                                         not
>>                                                                                         to
>>                                                                                         introduce
>>                                                                                         an
>>                                                                                         additional
>>                                                                                         SQL
>>                                                                                         syntax
>>                                                                                         to
>>                                                                                         set
>>                                                                                         the
>>                                                                                         default-catalog.
>>
>>                                                                                         Generally
>>                                                                                         we
>>                                                                                         have
>>                                                                                         different
>>                                                                                         environments
>>                                                                                         we
>>                                                                                         want
>>                                                                                         to
>>                                                                                         support
>>                                                                                         with
>>                                                                                         the
>>                                                                                         view
>>                                                                                         spec:
>>
>>                                                                                         1.
>>                                                                                         Consistent
>>                                                                                         catalog
>>                                                                                         naming
>>
>>                                                                                         When
>>                                                                                         the
>>                                                                                         environment
>>                                                                                         supports
>>                                                                                         it,
>>                                                                                         using
>>                                                                                         consistent
>>                                                                                         catalog
>>                                                                                         names
>>                                                                                         can
>>                                                                                         have
>>                                                                                         a
>>                                                                                         great
>>                                                                                         benefit
>>                                                                                         for
>>                                                                                         multi-catalog,
>>                                                                                         multi-engine
>>                                                                                         setups.
>>                                                                                         With
>>                                                                                         consistent
>>                                                                                         catalog
>>                                                                                         names,
>>                                                                                         using
>>                                                                                         the
>>                                                                                         "default-catalog"
>>                                                                                         field
>>                                                                                         works
>>                                                                                         without
>>                                                                                         any
>>                                                                                         issues.
>>
>>                                                                                         2.
>>                                                                                         Inconsistent
>>                                                                                         catalog
>>                                                                                         naming
>>
>>                                                                                         This
>>                                                                                         can
>>                                                                                         be
>>                                                                                         the
>>                                                                                         case
>>                                                                                         when
>>                                                                                         different
>>                                                                                         query
>>                                                                                         engines
>>                                                                                         refer
>>                                                                                         to
>>                                                                                         the
>>                                                                                         same
>>                                                                                         physical
>>                                                                                         catalog
>>                                                                                         by
>>                                                                                         different
>>                                                                                         names.
>>                                                                                         This
>>                                                                                         often
>>                                                                                         happens
>>                                                                                         because
>>                                                                                         different
>>                                                                                         query
>>                                                                                         engines
>>                                                                                         use
>>                                                                                         different
>>                                                                                         strategies
>>                                                                                         to
>>                                                                                         setup
>>                                                                                         the
>>                                                                                         catalogs.
>>                                                                                         If
>>                                                                                         catalogs
>>                                                                                         have
>>                                                                                         inconsistent
>>                                                                                         naming,
>>                                                                                         using
>>                                                                                         the
>>                                                                                         "default-catalog"
>>                                                                                         field
>>                                                                                         does
>>                                                                                         not
>>                                                                                         work
>>                                                                                         because
>>                                                                                         it
>>                                                                                         is
>>                                                                                         not
>>                                                                                         guaranteed
>>                                                                                         that
>>                                                                                         the
>>                                                                                         catalog
>>                                                                                         name
>>                                                                                         can
>>                                                                                         be
>>                                                                                         resolved
>>                                                                                         with
>>                                                                                         another
>>                                                                                         engine.
>>                                                                                         Using
>>                                                                                         the
>>                                                                                         "view
>>                                                                                         catalog"
>>                                                                                         as
>>                                                                                         a
>>                                                                                         fallback
>>                                                                                         is
>>                                                                                         a
>>                                                                                         better
>>                                                                                         solution
>>                                                                                         for
>>                                                                                         this
>>                                                                                         use
>>                                                                                         case,
>>                                                                                         as
>>                                                                                         it
>>                                                                                         avoids
>>                                                                                         catalog
>>                                                                                         names
>>                                                                                         altogether.
>>                                                                                         It
>>                                                                                         is
>>                                                                                         however
>>                                                                                         limited
>>                                                                                         to
>>                                                                                         table
>>                                                                                         references
>>                                                                                         in
>>                                                                                         the
>>                                                                                         same
>>                                                                                         catalog.
>>
>>
>>                                                                                         What
>>                                                                                         do
>>                                                                                         you
>>                                                                                         think
>>                                                                                         of
>>                                                                                         introducing
>>                                                                                         a
>>                                                                                         view
>>                                                                                         property
>>                                                                                         that
>>                                                                                         specifies
>>                                                                                         if
>>                                                                                         the
>>                                                                                         "default-catalog"
>>                                                                                         or
>>                                                                                         the
>>                                                                                         "view
>>                                                                                         catalog"
>>                                                                                         should
>>                                                                                         be
>>                                                                                         used?
>>                                                                                         This
>>                                                                                         way,
>>                                                                                         you
>>                                                                                         could
>>                                                                                         use
>>                                                                                         the
>>                                                                                         "default-catalog"
>>                                                                                         in
>>                                                                                         environments
>>                                                                                         where
>>                                                                                         you
>>                                                                                         can
>>                                                                                         guarantee
>>                                                                                         consistent
>>                                                                                         naming,
>>                                                                                         but
>>                                                                                         you
>>                                                                                         would
>>                                                                                         be
>>                                                                                         able
>>                                                                                         to
>>                                                                                         directly
>>                                                                                         fallback
>>                                                                                         to
>>                                                                                         the
>>                                                                                         "view-catalog"
>>                                                                                         when
>>                                                                                         you
>>                                                                                         don't
>>                                                                                         have
>>                                                                                         consistent
>>                                                                                         naming.
>>                                                                                         The
>>                                                                                         query
>>                                                                                         engines
>>                                                                                         could
>>                                                                                         set
>>                                                                                         the
>>                                                                                         default
>>                                                                                         for
>>                                                                                         this
>>                                                                                         view
>>                                                                                         property
>>                                                                                         at
>>                                                                                         creation
>>                                                                                         time.
>>                                                                                         Spark
>>                                                                                         for
>>                                                                                         example
>>                                                                                         could
>>                                                                                         set
>>                                                                                         it
>>                                                                                         to
>>                                                                                         automatically
>>                                                                                         use
>>                                                                                         the
>>                                                                                         "view
>>                                                                                         catalog".
>>
>>                                                                                         Thanks
>>
>>                                                                                         Jan
>>
>>
>>                                                                                         On
>>                                                                                         4/26/25
>>                                                                                         05:33,
>>                                                                                         Walaa
>>                                                                                         Eldin
>>                                                                                         Moustafa
>>                                                                                         wrote:
>>>                                                                                         To
>>>                                                                                         help
>>>                                                                                         folks
>>>                                                                                         catch
>>>                                                                                         up
>>>                                                                                         on
>>>                                                                                         the
>>>                                                                                         latest
>>>                                                                                         discussions
>>>                                                                                         and
>>>                                                                                         interpretation
>>>                                                                                         of
>>>                                                                                         the
>>>                                                                                         spec,
>>>                                                                                         I
>>>                                                                                         have
>>>                                                                                         summarized
>>>                                                                                         everything
>>>                                                                                         we
>>>                                                                                         discussed
>>>                                                                                         so
>>>                                                                                         far
>>>                                                                                         at
>>>                                                                                         the
>>>                                                                                         top
>>>                                                                                         of
>>>                                                                                         the
>>>                                                                                         proposal
>>>                                                                                         document
>>>                                                                                         (here
>>>                                                                                         <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>                                                                                         I
>>>                                                                                         have
>>>                                                                                         slightly
>>>                                                                                         updated
>>>                                                                                         the
>>>                                                                                         proposal
>>>                                                                                         to
>>>                                                                                         be
>>>                                                                                         in
>>>                                                                                         sync
>>>                                                                                         with
>>>                                                                                         the
>>>                                                                                         new
>>>                                                                                         interpretation
>>>                                                                                         to
>>>                                                                                         avoid
>>>                                                                                         confusion.
>>>                                                                                         In
>>>                                                                                         summary:
>>>
>>>                                                                                         *
>>>                                                                                         Remove
>>>                                                                                         default-catalog
>>>                                                                                         and
>>>                                                                                         default-namespace
>>>                                                                                         fields
>>>                                                                                         from
>>>                                                                                         the
>>>                                                                                         view
>>>                                                                                         spec
>>>                                                                                         completely.
>>>
>>>                                                                                         *
>>>                                                                                         Hence,
>>>                                                                                         we
>>>                                                                                         do
>>>                                                                                         not
>>>                                                                                         attempt
>>>                                                                                         to
>>>                                                                                         define
>>>                                                                                         separate
>>>                                                                                         view-level
>>>                                                                                         default
>>>                                                                                         catalogs
>>>                                                                                         or
>>>                                                                                         namespaces.
>>>
>>>                                                                                         Instead:
>>>
>>>                                                                                         *
>>>                                                                                         If
>>>                                                                                         a
>>>                                                                                         table
>>>                                                                                         identifier
>>>                                                                                         inside
>>>                                                                                         a
>>>                                                                                         view
>>>                                                                                         lacks
>>>                                                                                         a
>>>                                                                                         catalog
>>>                                                                                         qualifier,
>>>                                                                                         engines
>>>                                                                                         should
>>>                                                                                         resolve
>>>                                                                                         it
>>>                                                                                         using
>>>                                                                                         the
>>>                                                                                         current
>>>                                                                                         engine
>>>                                                                                         catalog
>>>                                                                                         at
>>>                                                                                         query
>>>                                                                                         time.
>>>
>>>                                                                                         *
>>>                                                                                         Reference
>>>                                                                                         table
>>>                                                                                         identifiers
>>>                                                                                         in
>>>                                                                                         the
>>>                                                                                         metadata
>>>                                                                                         exactly
>>>                                                                                         as
>>>                                                                                         they
>>>                                                                                         appear
>>>                                                                                         in
>>>                                                                                         the
>>>                                                                                         view
>>>                                                                                         SQL
>>>                                                                                         text.
>>>
>>>                                                                                         *
>>>                                                                                         If
>>>                                                                                         an
>>>                                                                                         identifier
>>>                                                                                         lacks
>>>                                                                                         the
>>>                                                                                         catalog
>>>                                                                                         part
>>>                                                                                         at
>>>                                                                                         creation,
>>>                                                                                         it
>>>                                                                                         should
>>>                                                                                         still
>>>                                                                                         lack
>>>                                                                                         a
>>>                                                                                         catalog
>>>                                                                                         in
>>>                                                                                         the
>>>                                                                                         stored
>>>                                                                                         metadata.
>>>
>>>                                                                                         *
>>>                                                                                         Store
>>>                                                                                         UUIDs
>>>                                                                                         alongside
>>>                                                                                         table
>>>                                                                                         identifiers
>>>                                                                                         whenever
>>>                                                                                         possible.
>>>
>>>                                                                                         Thanks,
>>>                                                                                         Walaa.
>>>
>>>
>>>                                                                                         On
>>>                                                                                         Fri,
>>>                                                                                         Apr
>>>                                                                                         25,
>>>                                                                                         2025
>>>                                                                                         at
>>>                                                                                         5:18 PM
>>>                                                                                         Walaa
>>>                                                                                         Eldin
>>>                                                                                         Moustafa
>>>                                                                                         <wa...@gmail.com>
>>>                                                                                         wrote:
>>>
>>>                                                                                             Thanks
>>>                                                                                             for
>>>                                                                                             the
>>>                                                                                             contribution
>>>                                                                                             Benny!
>>>                                                                                             +1
>>>                                                                                             to
>>>                                                                                             the
>>>                                                                                             confusion
>>>                                                                                             the
>>>                                                                                             fallback
>>>                                                                                             creates.
>>>                                                                                             Also
>>>                                                                                             just
>>>                                                                                             to
>>>                                                                                             be
>>>                                                                                             clear,
>>>                                                                                             at
>>>                                                                                             this
>>>                                                                                             point
>>>                                                                                             and
>>>                                                                                             after
>>>                                                                                             clarifying
>>>                                                                                             the
>>>                                                                                             current
>>>                                                                                             spec
>>>                                                                                             intentions,
>>>                                                                                             I
>>>                                                                                             am
>>>                                                                                             convinced
>>>                                                                                             that
>>>                                                                                             we
>>>                                                                                             should
>>>                                                                                             remove
>>>                                                                                             the
>>>                                                                                             default
>>>                                                                                             catalog
>>>                                                                                             and
>>>                                                                                             default
>>>                                                                                             namespace
>>>                                                                                             fields
>>>                                                                                             altogether.
>>>
>>>                                                                                             Thanks,
>>>                                                                                             Walaa.
>>>
>>>                                                                                             On
>>>                                                                                             Fri,
>>>                                                                                             Apr
>>>                                                                                             25,
>>>                                                                                             2025
>>>                                                                                             at
>>>                                                                                             5:13 PM
>>>                                                                                             Benny
>>>                                                                                             Chow
>>>                                                                                             <bt...@gmail.com>
>>>                                                                                             wrote:
>>>
>>>                                                                                                 I'd
>>>                                                                                                 like
>>>                                                                                                 to
>>>                                                                                                 contribute
>>>                                                                                                 my
>>>                                                                                                 opinions
>>>                                                                                                 on
>>>                                                                                                 this:
>>>
>>>                                                                                                 -
>>>                                                                                                 I
>>>                                                                                                 don't
>>>                                                                                                 particularly
>>>                                                                                                 like
>>>                                                                                                 the
>>>                                                                                                 current
>>>                                                                                                 behavior
>>>                                                                                                 of
>>>                                                                                                 "default
>>>                                                                                                 to
>>>                                                                                                 the
>>>                                                                                                 view's
>>>                                                                                                 catalog
>>>                                                                                                 when
>>>                                                                                                 default-catalog
>>>                                                                                                 is
>>>                                                                                                 not
>>>                                                                                                 set".
>>>                                                                                                 Fundamentally,
>>>                                                                                                 I
>>>                                                                                                 believe
>>>                                                                                                 the
>>>                                                                                                 intent
>>>                                                                                                 of
>>>                                                                                                 default-catalog
>>>                                                                                                 and
>>>                                                                                                 default-namespace
>>>                                                                                                 is
>>>                                                                                                 there
>>>                                                                                                 to
>>>                                                                                                 help
>>>                                                                                                 users
>>>                                                                                                 write
>>>                                                                                                 more
>>>                                                                                                 concise
>>>                                                                                                 SQL.
>>>                                                                                                 -
>>>                                                                                                 spark
>>>                                                                                                 session
>>>                                                                                                 catalog
>>>                                                                                                 is
>>>                                                                                                 engine
>>>                                                                                                 specific
>>>                                                                                                 and
>>>                                                                                                 I
>>>                                                                                                 don't
>>>                                                                                                 think
>>>                                                                                                 we
>>>                                                                                                 should
>>>                                                                                                 design
>>>                                                                                                 something
>>>                                                                                                 that
>>>                                                                                                 says
>>>                                                                                                 first
>>>                                                                                                 use
>>>                                                                                                 this
>>>                                                                                                 catalog,
>>>                                                                                                 then
>>>                                                                                                 that
>>>                                                                                                 catalog..
>>>                                                                                                 or
>>>                                                                                                 that
>>>                                                                                                 catalog.
>>>                                                                                                 For
>>>                                                                                                 example,
>>>                                                                                                 resolving
>>>                                                                                                 identifiers
>>>                                                                                                 using
>>>                                                                                                 default-catalog
>>>                                                                                                 ->
>>>                                                                                                 view's
>>>                                                                                                 catalog
>>>                                                                                                 ->
>>>                                                                                                 session
>>>                                                                                                 catalog
>>>                                                                                                 is
>>>                                                                                                 not
>>>                                                                                                 good.
>>>                                                                                                 -
>>>                                                                                                 We
>>>                                                                                                 gotta
>>>                                                                                                 support
>>>                                                                                                 non-Iceberg
>>>                                                                                                 tables
>>>                                                                                                 otherwise
>>>                                                                                                 I
>>>                                                                                                 see
>>>                                                                                                 no
>>>                                                                                                 value
>>>                                                                                                 in
>>>                                                                                                 putting
>>>                                                                                                 views
>>>                                                                                                 in
>>>                                                                                                 the
>>>                                                                                                 catalog
>>>                                                                                                 to
>>>                                                                                                 share
>>>                                                                                                 with
>>>                                                                                                 other
>>>                                                                                                 engines
>>>                                                                                                 -
>>>                                                                                                 Interoperability
>>>                                                                                                 between different engine
>>>                                                                                                 types
>>>                                                                                                 is
>>>                                                                                                 very
>>>                                                                                                 hard
>>>                                                                                                 due
>>>                                                                                                 to
>>>                                                                                                 dialect issues...
>>>                                                                                                 so
>>>                                                                                                 I
>>>                                                                                                 think
>>>                                                                                                 we
>>>                                                                                                 should focus
>>>                                                                                                 on
>>>                                                                                                 supporting
>>>                                                                                                 different
>>>                                                                                                 clusters
>>>                                                                                                 of
>>>                                                                                                 the
>>>                                                                                                 same
>>>                                                                                                 engine
>>>                                                                                                 type
>>>                                                                                                 on
>>>                                                                                                 a
>>>                                                                                                 shared
>>>                                                                                                 catalog. 
>>>                                                                                                 For
>>>                                                                                                 example,
>>>                                                                                                 AI
>>>                                                                                                 and
>>>                                                                                                 BI
>>>                                                                                                 clusters
>>>                                                                                                 on
>>>                                                                                                 Spark
>>>                                                                                                 sharing
>>>                                                                                                 the
>>>                                                                                                 same
>>>                                                                                                 views
>>>                                                                                                 in
>>>                                                                                                 a
>>>                                                                                                 REST
>>>                                                                                                 catalog.
>>>
>>>                                                                                                 Coincidentally,
>>>                                                                                                 I
>>>                                                                                                 think
>>>                                                                                                 the
>>>                                                                                                 ultimate
>>>                                                                                                 solution
>>>                                                                                                 is
>>>                                                                                                 along
>>>                                                                                                 the
>>>                                                                                                 lines
>>>                                                                                                 of
>>>                                                                                                 something
>>>                                                                                                 Russell
>>>                                                                                                 proposed
>>>                                                                                                 last
>>>                                                                                                 year:
>>>
>>>                                                                                                 https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>
>>>                                                                                                 We've
>>>                                                                                                 been
>>>                                                                                                 looking
>>>                                                                                                 at
>>>                                                                                                 this
>>>                                                                                                 interoperable
>>>                                                                                                 identifier
>>>                                                                                                 problem
>>>                                                                                                 through
>>>                                                                                                 the
>>>                                                                                                 lens
>>>                                                                                                 of
>>>                                                                                                 catalog
>>>                                                                                                 resolution
>>>                                                                                                 but
>>>                                                                                                 maybe
>>>                                                                                                 the
>>>                                                                                                 right
>>>                                                                                                 approach
>>>                                                                                                 is
>>>                                                                                                 really
>>>                                                                                                 about
>>>                                                                                                 templating.
>>>
>>>                                                                                                 I
>>>                                                                                                 would
>>>                                                                                                 extend
>>>                                                                                                 Russell's
>>>                                                                                                 idea
>>>                                                                                                 to
>>>                                                                                                 allow
>>>                                                                                                 identifiers
>>>                                                                                                 in
>>>                                                                                                 a
>>>                                                                                                 view
>>>                                                                                                 to
>>>                                                                                                 span
>>>                                                                                                 catalogs
>>>                                                                                                 to
>>>                                                                                                 support
>>>                                                                                                 non-Iceberg
>>>                                                                                                 tables.
>>>                                                                                                  Also,
>>>                                                                                                 the
>>>                                                                                                 default-catalog
>>>                                                                                                 property
>>>                                                                                                 could
>>>                                                                                                 be
>>>                                                                                                 templated
>>>                                                                                                 as
>>>                                                                                                 well.
>>>
>>>
>>>                                                                                                 Thoughts?
>>>                                                                                                 Benny
>>>
>>>
>>>
>>>                                                                                                 On
>>>                                                                                                 Fri,
>>>                                                                                                 Apr
>>>                                                                                                 25,
>>>                                                                                                 2025
>>>                                                                                                 at
>>>                                                                                                 4:02 PM
>>>                                                                                                 Walaa
>>>                                                                                                 Eldin
>>>                                                                                                 Moustafa
>>>                                                                                                 <wa...@gmail.com>
>>>                                                                                                 wrote:
>>>
>>>                                                                                                     Thanks
>>>                                                                                                     Steven!
>>>                                                                                                     How
>>>                                                                                                     do
>>>                                                                                                     you
>>>                                                                                                     recommend
>>>                                                                                                     making
>>>                                                                                                     Spark
>>>                                                                                                     implementation
>>>                                                                                                     conform
>>>                                                                                                     to
>>>                                                                                                     the
>>>                                                                                                     spec?
>>>                                                                                                     Do
>>>                                                                                                     we
>>>                                                                                                     need
>>>                                                                                                     Spark
>>>                                                                                                     SQL
>>>                                                                                                     extensions
>>>                                                                                                     and/or
>>>                                                                                                     Spark
>>>                                                                                                     catalog
>>>                                                                                                     APIs
>>>                                                                                                     for
>>>                                                                                                     that?
>>>
>>>                                                                                                     How
>>>                                                                                                     do
>>>                                                                                                     you
>>>                                                                                                     recommend
>>>                                                                                                     reconciling
>>>                                                                                                     the
>>>                                                                                                     inconsistencies
>>>                                                                                                     I
>>>                                                                                                     shared
>>>                                                                                                     regarding
>>>                                                                                                     many
>>>                                                                                                     resolution
>>>                                                                                                     methods
>>>                                                                                                     not
>>>                                                                                                     consistently
>>>                                                                                                     being
>>>                                                                                                     followed
>>>                                                                                                     in
>>>                                                                                                     different
>>>                                                                                                     scenarios
>>>                                                                                                     (view
>>>                                                                                                     vs
>>>                                                                                                     child
>>>                                                                                                     table
>>>                                                                                                     resolution,
>>>                                                                                                     query
>>>                                                                                                     vs
>>>                                                                                                     view
>>>                                                                                                     resolution)?
>>>                                                                                                     Note
>>>                                                                                                     these
>>>                                                                                                     occur
>>>                                                                                                     when
>>>                                                                                                     the
>>>                                                                                                     default
>>>                                                                                                     catalog
>>>                                                                                                     is
>>>                                                                                                     set
>>>                                                                                                     to
>>>                                                                                                     a
>>>                                                                                                     non-null
>>>                                                                                                     value.
>>>                                                                                                     If
>>>                                                                                                     it
>>>                                                                                                     helps,
>>>                                                                                                     I
>>>                                                                                                     can
>>>                                                                                                     share
>>>                                                                                                     concrete
>>>                                                                                                     examples.
>>>
>>>                                                                                                     Thanks,
>>>                                                                                                     Walaa.
>>>
>>>                                                                                                     On
>>>                                                                                                     Fri,
>>>                                                                                                     Apr
>>>                                                                                                     25,
>>>                                                                                                     2025
>>>                                                                                                     at
>>>                                                                                                     3:52 PM
>>>                                                                                                     Steven
>>>                                                                                                     Wu
>>>                                                                                                     <st...@gmail.com>
>>>                                                                                                     wrote:
>>>
>>>                                                                                                         The
>>>                                                                                                         core
>>>                                                                                                         issue
>>>                                                                                                         is
>>>                                                                                                         on
>>>                                                                                                         the
>>>                                                                                                         fall
>>>                                                                                                         back
>>>                                                                                                         behavior
>>>                                                                                                         when
>>>                                                                                                         `default-catalog`
>>>                                                                                                         is
>>>                                                                                                         not
>>>                                                                                                         defined.
>>>                                                                                                         Current
>>>                                                                                                         view
>>>                                                                                                         spec
>>>                                                                                                         says
>>>                                                                                                         the
>>>                                                                                                         fallback
>>>                                                                                                         should
>>>                                                                                                         be
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         where
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         defined.
>>>                                                                                                         It
>>>                                                                                                         doesn't
>>>                                                                                                         really
>>>                                                                                                         matter
>>>                                                                                                         what
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         is
>>>                                                                                                         named
>>>                                                                                                         (catalogX)
>>>                                                                                                         by
>>>                                                                                                         the
>>>                                                                                                         read
>>>                                                                                                         engine.
>>>                                                                                                         -
>>>                                                                                                         If
>>>                                                                                                         a
>>>                                                                                                         view
>>>                                                                                                         refers
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         tables
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         catalog,
>>>                                                                                                         this
>>>                                                                                                         is
>>>                                                                                                         a
>>>                                                                                                         non-ambiguous
>>>                                                                                                         and
>>>                                                                                                         reasonable
>>>                                                                                                         fallback
>>>                                                                                                         behavior.
>>>                                                                                                         -
>>>                                                                                                         If
>>>                                                                                                         a
>>>                                                                                                         view
>>>                                                                                                         refers
>>>                                                                                                         to
>>>                                                                                                         tables
>>>                                                                                                         from
>>>                                                                                                         another
>>>                                                                                                         catalog,
>>>                                                                                                         catalog
>>>                                                                                                         names
>>>                                                                                                         should
>>>                                                                                                         be
>>>                                                                                                         included
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         reference
>>>                                                                                                         name
>>>                                                                                                         already.
>>>                                                                                                         So
>>>                                                                                                         no
>>>                                                                                                         ambiguity
>>>                                                                                                         there
>>>                                                                                                         either.
>>>
>>>                                                                                                         Potential
>>>                                                                                                         inconsistent
>>>                                                                                                         naming
>>>                                                                                                         of
>>>                                                                                                         catalog
>>>                                                                                                         is
>>>                                                                                                         a
>>>                                                                                                         separate
>>>                                                                                                         problem,
>>>                                                                                                         which
>>>                                                                                                         Iceberg
>>>                                                                                                         view
>>>                                                                                                         spec
>>>                                                                                                         probably
>>>                                                                                                         cannot
>>>                                                                                                         solve.
>>>                                                                                                         We
>>>                                                                                                         can
>>>                                                                                                         only
>>>                                                                                                         recommend
>>>                                                                                                         that
>>>                                                                                                         catalog
>>>                                                                                                         should
>>>                                                                                                         be
>>>                                                                                                         named
>>>                                                                                                         consistently
>>>                                                                                                         across
>>>                                                                                                         usage
>>>                                                                                                         for
>>>                                                                                                         better
>>>                                                                                                         interoperability
>>>                                                                                                         on
>>>                                                                                                         name
>>>                                                                                                         references.
>>>
>>>                                                                                                         This
>>>                                                                                                         proposal
>>>                                                                                                         is
>>>                                                                                                         to
>>>                                                                                                         change
>>>                                                                                                         the
>>>                                                                                                         fallback
>>>                                                                                                         behavior
>>>                                                                                                         to
>>>                                                                                                         engine's
>>>                                                                                                         session
>>>                                                                                                         default
>>>                                                                                                         catalog.
>>>                                                                                                         I
>>>                                                                                                         am
>>>                                                                                                         not
>>>                                                                                                         sure
>>>                                                                                                         it
>>>                                                                                                         is
>>>                                                                                                         better
>>>                                                                                                         than
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         fallback
>>>                                                                                                         behavior.
>>>
>>>                                                                                                         >
>>>                                                                                                         Today’s
>>>                                                                                                         Spark
>>>                                                                                                         behavior
>>>                                                                                                         explicitly
>>>                                                                                                         differs
>>>                                                                                                         from
>>>                                                                                                         this
>>>                                                                                                         idea.
>>>                                                                                                         Spark
>>>                                                                                                         resolves
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         during
>>>                                                                                                         view
>>>                                                                                                         creation
>>>                                                                                                         using
>>>                                                                                                         the
>>>                                                                                                         session’s
>>>                                                                                                         default
>>>                                                                                                         catalog,
>>>                                                                                                         not
>>>                                                                                                         a
>>>                                                                                                         supplied
>>>                                                                                                         `default-catalog`.
>>>
>>>                                                                                                         I
>>>                                                                                                         would
>>>                                                                                                         argue
>>>                                                                                                         that
>>>                                                                                                         is
>>>                                                                                                         a
>>>                                                                                                         Spark
>>>                                                                                                         implementation
>>>                                                                                                         issue
>>>                                                                                                         for
>>>                                                                                                         not
>>>                                                                                                         conforming
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         spec.
>>>
>>>
>>>                                                                                                         On
>>>                                                                                                         Fri,
>>>                                                                                                         Apr
>>>                                                                                                         25,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         1:17 PM
>>>                                                                                                         Walaa
>>>                                                                                                         Eldin
>>>                                                                                                         Moustafa
>>>                                                                                                         <wa...@gmail.com>
>>>                                                                                                         wrote:
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         Hi
>>>                                                                                                         Jan,
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         Thanks
>>>                                                                                                         again
>>>                                                                                                         for
>>>                                                                                                         continuing
>>>                                                                                                         the
>>>                                                                                                         discussion.
>>>                                                                                                         I
>>>                                                                                                         want
>>>                                                                                                         to
>>>                                                                                                         highlight
>>>                                                                                                         a
>>>                                                                                                         few
>>>                                                                                                         fundamental
>>>                                                                                                         issues
>>>                                                                                                         around
>>>                                                                                                         the
>>>                                                                                                         interpretation
>>>                                                                                                         of
>>>                                                                                                         default-catalog:
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         Here
>>>                                                                                                         is
>>>                                                                                                         the
>>>                                                                                                         real
>>>                                                                                                         catch:
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         *
>>>                                                                                                         default-catalog
>>>                                                                                                         cannot
>>>                                                                                                         logically
>>>                                                                                                         be
>>>                                                                                                         defined
>>>                                                                                                         at
>>>                                                                                                         view
>>>                                                                                                         creation
>>>                                                                                                         time.
>>>                                                                                                         It
>>>                                                                                                         would
>>>                                                                                                         be
>>>                                                                                                         circular:
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         needs
>>>                                                                                                         to
>>>                                                                                                         exist
>>>                                                                                                         before
>>>                                                                                                         its
>>>                                                                                                         metadata
>>>                                                                                                         (and
>>>                                                                                                         hence
>>>                                                                                                         default-catalog)
>>>                                                                                                         can
>>>                                                                                                         exist.
>>>                                                                                                         This
>>>                                                                                                         is
>>>                                                                                                         visible
>>>                                                                                                         in
>>>                                                                                                         Spark’s
>>>                                                                                                         implementation,
>>>                                                                                                         where
>>>                                                                                                         `default-catalog`
>>>                                                                                                         is
>>>                                                                                                         not
>>>                                                                                                         used.
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         *
>>>                                                                                                         Introducing
>>>                                                                                                         a
>>>                                                                                                         creation-time
>>>                                                                                                         default-catalog
>>>                                                                                                         setting
>>>                                                                                                         would
>>>                                                                                                         require
>>>                                                                                                         extending
>>>                                                                                                         SQL
>>>                                                                                                         syntax
>>>                                                                                                         and
>>>                                                                                                         engine
>>>                                                                                                         APIs
>>>                                                                                                         to
>>>                                                                                                         promote
>>>                                                                                                         it
>>>                                                                                                         to
>>>                                                                                                         a
>>>                                                                                                         first-class
>>>                                                                                                         view
>>>                                                                                                         concept.
>>>                                                                                                         This
>>>                                                                                                         would
>>>                                                                                                         be
>>>                                                                                                         intrusive,
>>>                                                                                                         non-intuitive,
>>>                                                                                                         and
>>>                                                                                                         realistically
>>>                                                                                                         very
>>>                                                                                                         difficult
>>>                                                                                                         to
>>>                                                                                                         standardize
>>>                                                                                                         across
>>>                                                                                                         engines.
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         *
>>>                                                                                                         Today’s
>>>                                                                                                         Spark
>>>                                                                                                         behavior
>>>                                                                                                         explicitly
>>>                                                                                                         differs
>>>                                                                                                         from
>>>                                                                                                         this
>>>                                                                                                         idea.
>>>                                                                                                         Spark
>>>                                                                                                         resolves
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         during
>>>                                                                                                         view
>>>                                                                                                         creation
>>>                                                                                                         using
>>>                                                                                                         the
>>>                                                                                                         session’s
>>>                                                                                                         default
>>>                                                                                                         catalog,
>>>                                                                                                         not
>>>                                                                                                         a
>>>                                                                                                         supplied
>>>                                                                                                         `default-catalog`.
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         *
>>>                                                                                                         Hypothetically
>>>                                                                                                         even
>>>                                                                                                         if
>>>                                                                                                         we
>>>                                                                                                         patched
>>>                                                                                                         in
>>>                                                                                                         a
>>>                                                                                                         creation-time
>>>                                                                                                         default-catalog,
>>>                                                                                                         it
>>>                                                                                                         would
>>>                                                                                                         create
>>>                                                                                                         an
>>>                                                                                                         inconsistent
>>>                                                                                                         binding
>>>                                                                                                         model
>>>                                                                                                         between
>>>                                                                                                         tables
>>>                                                                                                         vs
>>>                                                                                                         views
>>>                                                                                                         (early
>>>                                                                                                         vs
>>>                                                                                                         late),
>>>                                                                                                         and
>>>                                                                                                         between
>>>                                                                                                         tables
>>>                                                                                                         in
>>>                                                                                                         views
>>>                                                                                                         and
>>>                                                                                                         in
>>>                                                                                                         queries
>>>                                                                                                         (again
>>>                                                                                                         early
>>>                                                                                                         vs
>>>                                                                                                         late).
>>>                                                                                                         For
>>>                                                                                                         example,
>>>                                                                                                         views
>>>                                                                                                         and
>>>                                                                                                         tables
>>>                                                                                                         in
>>>                                                                                                         queries
>>>                                                                                                         can
>>>                                                                                                         withstand
>>>                                                                                                         default
>>>                                                                                                         catalog
>>>                                                                                                         renames,
>>>                                                                                                         but
>>>                                                                                                         tables
>>>                                                                                                         cannot
>>>                                                                                                         when
>>>                                                                                                         they
>>>                                                                                                         are
>>>                                                                                                         used
>>>                                                                                                         inside
>>>                                                                                                         views
>>>                                                                                                         --
>>>                                                                                                         it
>>>                                                                                                         even
>>>                                                                                                         applies
>>>                                                                                                         to
>>>                                                                                                         views
>>>                                                                                                         inside
>>>                                                                                                         views,
>>>                                                                                                         which
>>>                                                                                                         makes
>>>                                                                                                         this
>>>                                                                                                         very
>>>                                                                                                         hard
>>>                                                                                                         to
>>>                                                                                                         reason
>>>                                                                                                         about
>>>                                                                                                         considering
>>>                                                                                                         nesting.
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         Thanks,
>>>                                                                                                         >
>>>                                                                                                         Walaa
>>>                                                                                                         >
>>>                                                                                                         >
>>>                                                                                                         On
>>>                                                                                                         Fri,
>>>                                                                                                         Apr
>>>                                                                                                         25,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         7:00 AM
>>>                                                                                                         Jan
>>>                                                                                                         Kaul
>>>                                                                                                         <ja...@mailbox.org.invalid>
>>>                                                                                                         <ma...@mailbox.org.invalid>
>>>                                                                                                         wrote:
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         @Walaa:
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         I
>>>                                                                                                         would
>>>                                                                                                         argue
>>>                                                                                                         that
>>>                                                                                                         when
>>>                                                                                                         you
>>>                                                                                                         run
>>>                                                                                                         a
>>>                                                                                                         CREATE
>>>                                                                                                         VIEW
>>>                                                                                                         statement
>>>                                                                                                         the
>>>                                                                                                         query
>>>                                                                                                         engine
>>>                                                                                                         knowns
>>>                                                                                                         which
>>>                                                                                                         catalog
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         being
>>>                                                                                                         created
>>>                                                                                                         in.
>>>                                                                                                         So
>>>                                                                                                         even
>>>                                                                                                         though
>>>                                                                                                         we
>>>                                                                                                         typically
>>>                                                                                                         use
>>>                                                                                                         late
>>>                                                                                                         binding
>>>                                                                                                         to
>>>                                                                                                         resolve
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         catalog
>>>                                                                                                         at
>>>                                                                                                         query
>>>                                                                                                         time,
>>>                                                                                                         it
>>>                                                                                                         can
>>>                                                                                                         also
>>>                                                                                                         be
>>>                                                                                                         used
>>>                                                                                                         at
>>>                                                                                                         creation
>>>                                                                                                         time.
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         The
>>>                                                                                                         query
>>>                                                                                                         engine
>>>                                                                                                         would
>>>                                                                                                         need
>>>                                                                                                         to
>>>                                                                                                         keep
>>>                                                                                                         track
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         "view
>>>                                                                                                         catalog"
>>>                                                                                                         where
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         going
>>>                                                                                                         to
>>>                                                                                                         be
>>>                                                                                                         created
>>>                                                                                                         in.
>>>                                                                                                         It
>>>                                                                                                         can
>>>                                                                                                         use
>>>                                                                                                         that
>>>                                                                                                         catalog
>>>                                                                                                         to
>>>                                                                                                         resolve
>>>                                                                                                         partial
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         if
>>>                                                                                                         "default-catalog"
>>>                                                                                                         is
>>>                                                                                                         not
>>>                                                                                                         set.
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         It
>>>                                                                                                         can
>>>                                                                                                         lead
>>>                                                                                                         to
>>>                                                                                                         some
>>>                                                                                                         unintuitive
>>>                                                                                                         behavior,
>>>                                                                                                         where
>>>                                                                                                         partial
>>>                                                                                                         identifiers
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         query
>>>                                                                                                         resolve
>>>                                                                                                         to
>>>                                                                                                         a
>>>                                                                                                         different
>>>                                                                                                         catalog
>>>                                                                                                         compared
>>>                                                                                                         to
>>>                                                                                                         using
>>>                                                                                                         them
>>>                                                                                                         outside
>>>                                                                                                         of
>>>                                                                                                         a
>>>                                                                                                         view.
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         CREATE
>>>                                                                                                         VIEW
>>>                                                                                                         catalogA.sales.monthly_orders
>>>                                                                                                         AS
>>>                                                                                                         SELECT
>>>                                                                                                         *
>>>                                                                                                         from
>>>                                                                                                         sales.orders;
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         If
>>>                                                                                                         the
>>>                                                                                                         session
>>>                                                                                                         default
>>>                                                                                                         catalog
>>>                                                                                                         is
>>>                                                                                                         not
>>>                                                                                                         "catalogA",
>>>                                                                                                         the
>>>                                                                                                         "sales.orders"
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         query
>>>                                                                                                         would
>>>                                                                                                         not
>>>                                                                                                         be
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         as
>>>                                                                                                         just
>>>                                                                                                         referencing
>>>                                                                                                         "sales.orders"
>>>                                                                                                         in
>>>                                                                                                         a
>>>                                                                                                         normal
>>>                                                                                                         SQL
>>>                                                                                                         statement.
>>>                                                                                                         This
>>>                                                                                                         is
>>>                                                                                                         because
>>>                                                                                                         without
>>>                                                                                                         a
>>>                                                                                                         "default-catalog",
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         name
>>>                                                                                                         of
>>>                                                                                                         "sales.orders"
>>>                                                                                                         would
>>>                                                                                                         default
>>>                                                                                                         to
>>>                                                                                                         "catalogA",
>>>                                                                                                         which
>>>                                                                                                         is
>>>                                                                                                         the
>>>                                                                                                         view's
>>>                                                                                                         catalog.
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         Jan
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         On
>>>                                                                                                         4/25/25
>>>                                                                                                         04:05,
>>>                                                                                                         Manu
>>>                                                                                                         Zhang
>>>                                                                                                         wrote:
>>>                                                                                                         >>>
>>>                                                                                                         >>>
>>>                                                                                                         For
>>>                                                                                                         example,
>>>                                                                                                         if
>>>                                                                                                         we
>>>                                                                                                         want
>>>                                                                                                         to
>>>                                                                                                         validate
>>>                                                                                                         that
>>>                                                                                                         the
>>>                                                                                                         tables
>>>                                                                                                         referenced
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         exist,
>>>                                                                                                         how
>>>                                                                                                         can
>>>                                                                                                         we
>>>                                                                                                         do
>>>                                                                                                         that
>>>                                                                                                         when
>>>                                                                                                         default-catalog
>>>                                                                                                         isn't
>>>                                                                                                         defined,
>>>                                                                                                         since
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         hasn't
>>>                                                                                                         been
>>>                                                                                                         created
>>>                                                                                                         or
>>>                                                                                                         loaded
>>>                                                                                                         yet?
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         I
>>>                                                                                                         don't
>>>                                                                                                         think
>>>                                                                                                         this
>>>                                                                                                         is
>>>                                                                                                         related
>>>                                                                                                         to
>>>                                                                                                         view
>>>                                                                                                         spec.
>>>                                                                                                         How
>>>                                                                                                         do
>>>                                                                                                         we
>>>                                                                                                         validate
>>>                                                                                                         that
>>>                                                                                                         a
>>>                                                                                                         table
>>>                                                                                                         exists
>>>                                                                                                         without
>>>                                                                                                         a
>>>                                                                                                         default
>>>                                                                                                         catalog,
>>>                                                                                                         or
>>>                                                                                                         do
>>>                                                                                                         we
>>>                                                                                                         always
>>>                                                                                                         use
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         session
>>>                                                                                                         catalog?
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>
>>>                                                                                                         Manu
>>>                                                                                                         >>
>>>                                                                                                         >>
>>>                                                                                                         On
>>>                                                                                                         Fri,
>>>                                                                                                         Apr
>>>                                                                                                         25,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         5:59 AM
>>>                                                                                                         Walaa
>>>                                                                                                         Eldin
>>>                                                                                                         Moustafa
>>>                                                                                                         <wa...@gmail.com>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>
>>>                                                                                                         >>>
>>>                                                                                                         Hi
>>>                                                                                                         Jan,
>>>                                                                                                         >>>
>>>                                                                                                         >>>
>>>                                                                                                         I
>>>                                                                                                         think
>>>                                                                                                         we
>>>                                                                                                         still
>>>                                                                                                         share
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         understanding.
>>>                                                                                                         Just
>>>                                                                                                         to
>>>                                                                                                         clarify:
>>>                                                                                                         when
>>>                                                                                                         I
>>>                                                                                                         referred
>>>                                                                                                         to
>>>                                                                                                         late
>>>                                                                                                         binding
>>>                                                                                                         as
>>>                                                                                                         “similar”
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         proposal,
>>>                                                                                                         I
>>>                                                                                                         was
>>>                                                                                                         acknowledging
>>>                                                                                                         the
>>>                                                                                                         distinction
>>>                                                                                                         between
>>>                                                                                                         view-level
>>>                                                                                                         and
>>>                                                                                                         table-level
>>>                                                                                                         resolution.
>>>                                                                                                         But
>>>                                                                                                         as
>>>                                                                                                         you
>>>                                                                                                         noted,
>>>                                                                                                         both
>>>                                                                                                         follow
>>>                                                                                                         a
>>>                                                                                                         late
>>>                                                                                                         binding
>>>                                                                                                         model.
>>>                                                                                                         >>>
>>>                                                                                                         >>>
>>>                                                                                                         That
>>>                                                                                                         said,
>>>                                                                                                         this
>>>                                                                                                         still
>>>                                                                                                         raises
>>>                                                                                                         an
>>>                                                                                                         interesting
>>>                                                                                                         question
>>>                                                                                                         and
>>>                                                                                                         a
>>>                                                                                                         potential
>>>                                                                                                         gap:
>>>                                                                                                         if
>>>                                                                                                         default-catalog
>>>                                                                                                         is
>>>                                                                                                         only
>>>                                                                                                         defined
>>>                                                                                                         at
>>>                                                                                                         query
>>>                                                                                                         time,
>>>                                                                                                         how
>>>                                                                                                         should
>>>                                                                                                         resolution
>>>                                                                                                         work
>>>                                                                                                         during
>>>                                                                                                         view
>>>                                                                                                         creation?
>>>                                                                                                         For
>>>                                                                                                         example,
>>>                                                                                                         if
>>>                                                                                                         we
>>>                                                                                                         want
>>>                                                                                                         to
>>>                                                                                                         validate
>>>                                                                                                         that
>>>                                                                                                         the
>>>                                                                                                         tables
>>>                                                                                                         referenced
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         exist,
>>>                                                                                                         how
>>>                                                                                                         can
>>>                                                                                                         we
>>>                                                                                                         do
>>>                                                                                                         that
>>>                                                                                                         when
>>>                                                                                                         default-catalog
>>>                                                                                                         isn't
>>>                                                                                                         defined,
>>>                                                                                                         since
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         hasn't
>>>                                                                                                         been
>>>                                                                                                         created
>>>                                                                                                         or
>>>                                                                                                         loaded
>>>                                                                                                         yet?
>>>                                                                                                         >>>
>>>                                                                                                         >>>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>>
>>>                                                                                                         Walaa.
>>>                                                                                                         >>>
>>>                                                                                                         >>>
>>>                                                                                                         On
>>>                                                                                                         Thu,
>>>                                                                                                         Apr
>>>                                                                                                         24,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         7:02 AM
>>>                                                                                                         Jan
>>>                                                                                                         Kaul
>>>                                                                                                         <ja...@mailbox.org.invalid>
>>>                                                                                                         <ma...@mailbox.org.invalid>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         Yes,
>>>                                                                                                         I
>>>                                                                                                         have
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         understanding.
>>>                                                                                                         The
>>>                                                                                                         view
>>>                                                                                                         catalog
>>>                                                                                                         is
>>>                                                                                                         resolved
>>>                                                                                                         at
>>>                                                                                                         query
>>>                                                                                                         time.
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         As
>>>                                                                                                         you
>>>                                                                                                         mentioned
>>>                                                                                                         before,
>>>                                                                                                         it's
>>>                                                                                                         good
>>>                                                                                                         to
>>>                                                                                                         distinguish
>>>                                                                                                         between
>>>                                                                                                         the
>>>                                                                                                         physical
>>>                                                                                                         catalog
>>>                                                                                                         and
>>>                                                                                                         it's
>>>                                                                                                         reference
>>>                                                                                                         used
>>>                                                                                                         in
>>>                                                                                                         SQL
>>>                                                                                                         statements.
>>>                                                                                                         The
>>>                                                                                                         important
>>>                                                                                                         part
>>>                                                                                                         is
>>>                                                                                                         that
>>>                                                                                                         the
>>>                                                                                                         physical
>>>                                                                                                         catalog
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         and
>>>                                                                                                         the
>>>                                                                                                         tables
>>>                                                                                                         referenced
>>>                                                                                                         in
>>>                                                                                                         it's
>>>                                                                                                         definition
>>>                                                                                                         stay
>>>                                                                                                         consistent.
>>>                                                                                                         You
>>>                                                                                                         could
>>>                                                                                                         create
>>>                                                                                                         a
>>>                                                                                                         view
>>>                                                                                                         in
>>>                                                                                                         a
>>>                                                                                                         given
>>>                                                                                                         physical
>>>                                                                                                         catalog
>>>                                                                                                         by
>>>                                                                                                         referring
>>>                                                                                                         to
>>>                                                                                                         it
>>>                                                                                                         as
>>>                                                                                                         "catalogA",
>>>                                                                                                         as
>>>                                                                                                         in
>>>                                                                                                         your
>>>                                                                                                         first
>>>                                                                                                         point.
>>>                                                                                                         If
>>>                                                                                                         you
>>>                                                                                                         then,
>>>                                                                                                         given
>>>                                                                                                         a
>>>                                                                                                         different
>>>                                                                                                         setup,
>>>                                                                                                         refer
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         physical
>>>                                                                                                         catalog
>>>                                                                                                         as
>>>                                                                                                         "catalogB"
>>>                                                                                                         in
>>>                                                                                                         another
>>>                                                                                                         session/environment,
>>>                                                                                                         the
>>>                                                                                                         behavior
>>>                                                                                                         should
>>>                                                                                                         still
>>>                                                                                                         work.
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         I
>>>                                                                                                         would
>>>                                                                                                         however
>>>                                                                                                         rephrase
>>>                                                                                                         your
>>>                                                                                                         last
>>>                                                                                                         point.
>>>                                                                                                         Late
>>>                                                                                                         binding
>>>                                                                                                         applies
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         catalog
>>>                                                                                                         name
>>>                                                                                                         and
>>>                                                                                                         by
>>>                                                                                                         extension
>>>                                                                                                         to
>>>                                                                                                         all
>>>                                                                                                         partial
>>>                                                                                                         table
>>>                                                                                                         references
>>>                                                                                                         when
>>>                                                                                                         no
>>>                                                                                                         "default-catalog"
>>>                                                                                                         is
>>>                                                                                                         present.
>>>                                                                                                         Resolving
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         catalog
>>>                                                                                                         name
>>>                                                                                                         at
>>>                                                                                                         query
>>>                                                                                                         time
>>>                                                                                                         is
>>>                                                                                                         not
>>>                                                                                                         opposed
>>>                                                                                                         to
>>>                                                                                                         storing
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         metadata
>>>                                                                                                         in
>>>                                                                                                         a
>>>                                                                                                         catalog.
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         Or
>>>                                                                                                         maybe
>>>                                                                                                         I
>>>                                                                                                         don't
>>>                                                                                                         entirely
>>>                                                                                                         understand
>>>                                                                                                         what
>>>                                                                                                         you
>>>                                                                                                         mean.
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         Thanks
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         Jan
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         On
>>>                                                                                                         4/24/25
>>>                                                                                                         00:32,
>>>                                                                                                         Walaa
>>>                                                                                                         Eldin
>>>                                                                                                         Moustafa
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         Hi
>>>                                                                                                         Jan,
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         >
>>>                                                                                                         The
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         executed
>>>                                                                                                         when
>>>                                                                                                         it's
>>>                                                                                                         being
>>>                                                                                                         referenced
>>>                                                                                                         in
>>>                                                                                                         a
>>>                                                                                                         SQL
>>>                                                                                                         statement.
>>>                                                                                                         That
>>>                                                                                                         statement
>>>                                                                                                         contains
>>>                                                                                                         the
>>>                                                                                                         information
>>>                                                                                                         for
>>>                                                                                                         the
>>>                                                                                                         query
>>>                                                                                                         engine
>>>                                                                                                         to
>>>                                                                                                         resolve
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         view.
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         If
>>>                                                                                                         I’m
>>>                                                                                                         understanding
>>>                                                                                                         correctly,
>>>                                                                                                         that
>>>                                                                                                         means:
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         *
>>>                                                                                                         If
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         queried
>>>                                                                                                         as
>>>                                                                                                         SELECT
>>>                                                                                                         *
>>>                                                                                                         FROM
>>>                                                                                                         catalogA.namespace.view,
>>>                                                                                                         then
>>>                                                                                                         catalogA
>>>                                                                                                         is
>>>                                                                                                         considered
>>>                                                                                                         the
>>>                                                                                                         view’s
>>>                                                                                                         catalog.
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         *
>>>                                                                                                         If
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         later
>>>                                                                                                         queried
>>>                                                                                                         as
>>>                                                                                                         SELECT
>>>                                                                                                         *
>>>                                                                                                         FROM
>>>                                                                                                         catalogB.namespace.view
>>>                                                                                                         (after
>>>                                                                                                         renaming
>>>                                                                                                         catalogA
>>>                                                                                                         to
>>>                                                                                                         catalogB,
>>>                                                                                                         and
>>>                                                                                                         keeping
>>>                                                                                                         everything
>>>                                                                                                         else
>>>                                                                                                         the
>>>                                                                                                         same),
>>>                                                                                                         then
>>>                                                                                                         catalogB
>>>                                                                                                         becomes
>>>                                                                                                         the
>>>                                                                                                         view’s
>>>                                                                                                         catalog.
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         Is
>>>                                                                                                         that
>>>                                                                                                         interpretation
>>>                                                                                                         correct?
>>>                                                                                                         If
>>>                                                                                                         so,
>>>                                                                                                         it
>>>                                                                                                         sounds
>>>                                                                                                         to
>>>                                                                                                         me
>>>                                                                                                         like
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         is
>>>                                                                                                         resolved
>>>                                                                                                         at
>>>                                                                                                         query
>>>                                                                                                         time,
>>>                                                                                                         based
>>>                                                                                                         on
>>>                                                                                                         how
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         referenced,
>>>                                                                                                         not
>>>                                                                                                         from
>>>                                                                                                         any
>>>                                                                                                         stored
>>>                                                                                                         metadata.
>>>                                                                                                         That
>>>                                                                                                         would
>>>                                                                                                         imply
>>>                                                                                                         some
>>>                                                                                                         sort
>>>                                                                                                         of
>>>                                                                                                         a
>>>                                                                                                         late
>>>                                                                                                         binding
>>>                                                                                                         behavior
>>>                                                                                                         (similar
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         proposal),
>>>                                                                                                         as
>>>                                                                                                         opposed
>>>                                                                                                         to
>>>                                                                                                         using
>>>                                                                                                         some
>>>                                                                                                         catalog
>>>                                                                                                         that
>>>                                                                                                         "stores"
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         definition.
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>>>
>>>                                                                                                         Walaa
>>>                                                                                                         >>>>
>>>                                                                                                         >>>>
>>>                                                                                                         On
>>>                                                                                                         Tue,
>>>                                                                                                         Apr
>>>                                                                                                         22,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         11:01 AM
>>>                                                                                                         Jan
>>>                                                                                                         Kaul
>>>                                                                                                         <ja...@mailbox.org.invalid>
>>>                                                                                                         <ma...@mailbox.org.invalid>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         Hi
>>>                                                                                                         Walaa,
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         Thanks
>>>                                                                                                         for
>>>                                                                                                         clarifying
>>>                                                                                                         the
>>>                                                                                                         aspects
>>>                                                                                                         of
>>>                                                                                                         non-determinism.
>>>                                                                                                         Let
>>>                                                                                                         me
>>>                                                                                                         try
>>>                                                                                                         to
>>>                                                                                                         address
>>>                                                                                                         your
>>>                                                                                                         questions.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         1.
>>>                                                                                                         This
>>>                                                                                                         is
>>>                                                                                                         my
>>>                                                                                                         interpretation
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         spec:
>>>                                                                                                         The
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         executed
>>>                                                                                                         when
>>>                                                                                                         it's
>>>                                                                                                         being
>>>                                                                                                         referenced
>>>                                                                                                         in
>>>                                                                                                         a
>>>                                                                                                         SQL
>>>                                                                                                         statement.
>>>                                                                                                         That
>>>                                                                                                         statement
>>>                                                                                                         contains
>>>                                                                                                         the
>>>                                                                                                         information
>>>                                                                                                         for
>>>                                                                                                         the
>>>                                                                                                         query
>>>                                                                                                         engine
>>>                                                                                                         to
>>>                                                                                                         resolve
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         view.
>>>                                                                                                         The
>>>                                                                                                         query
>>>                                                                                                         engine
>>>                                                                                                         then
>>>                                                                                                         uses
>>>                                                                                                         that
>>>                                                                                                         information
>>>                                                                                                         to
>>>                                                                                                         fetch
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         metadata
>>>                                                                                                         from
>>>                                                                                                         the
>>>                                                                                                         catalog.
>>>                                                                                                         It
>>>                                                                                                         also
>>>                                                                                                         needs
>>>                                                                                                         to
>>>                                                                                                         temporarily
>>>                                                                                                         keep
>>>                                                                                                         track
>>>                                                                                                         of
>>>                                                                                                         which
>>>                                                                                                         catalog
>>>                                                                                                         it
>>>                                                                                                         used
>>>                                                                                                         to
>>>                                                                                                         fetch
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         metadata.
>>>                                                                                                         It
>>>                                                                                                         can
>>>                                                                                                         then
>>>                                                                                                         use
>>>                                                                                                         that
>>>                                                                                                         information
>>>                                                                                                         to
>>>                                                                                                         resolve
>>>                                                                                                         the
>>>                                                                                                         table
>>>                                                                                                         references
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         views
>>>                                                                                                         SQL
>>>                                                                                                         definition
>>>                                                                                                         in
>>>                                                                                                         case
>>>                                                                                                         no
>>>                                                                                                         default
>>>                                                                                                         catalog
>>>                                                                                                         is
>>>                                                                                                         specified.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         2.
>>>                                                                                                         The
>>>                                                                                                         important
>>>                                                                                                         part
>>>                                                                                                         is
>>>                                                                                                         that
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         can
>>>                                                                                                         be
>>>                                                                                                         referenced
>>>                                                                                                         at
>>>                                                                                                         execution
>>>                                                                                                         time.
>>>                                                                                                         As
>>>                                                                                                         long
>>>                                                                                                         as
>>>                                                                                                         that's
>>>                                                                                                         the
>>>                                                                                                         case
>>>                                                                                                         I
>>>                                                                                                         would
>>>                                                                                                         assume
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         can
>>>                                                                                                         be
>>>                                                                                                         created
>>>                                                                                                         in
>>>                                                                                                         any
>>>                                                                                                         catalog.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         I
>>>                                                                                                         think
>>>                                                                                                         your
>>>                                                                                                         point
>>>                                                                                                         is
>>>                                                                                                         really
>>>                                                                                                         valuable
>>>                                                                                                         because
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         specification
>>>                                                                                                         can
>>>                                                                                                         lead
>>>                                                                                                         to
>>>                                                                                                         some
>>>                                                                                                         unintuitive
>>>                                                                                                         behavior.
>>>                                                                                                         For
>>>                                                                                                         example
>>>                                                                                                         for
>>>                                                                                                         the
>>>                                                                                                         following
>>>                                                                                                         statement:
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         CREATE
>>>                                                                                                         VIEW
>>>                                                                                                         catalogA.sales.monthly_orders
>>>                                                                                                         AS
>>>                                                                                                         SELECT
>>>                                                                                                         *
>>>                                                                                                         from
>>>                                                                                                         sales.orders;
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         If
>>>                                                                                                         the
>>>                                                                                                         session
>>>                                                                                                         default
>>>                                                                                                         catalog
>>>                                                                                                         is
>>>                                                                                                         not
>>>                                                                                                         "catalogA",
>>>                                                                                                         the
>>>                                                                                                         "sales.orders"
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         query
>>>                                                                                                         would
>>>                                                                                                         not
>>>                                                                                                         be
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         as
>>>                                                                                                         just
>>>                                                                                                         referencing
>>>                                                                                                         "sales.orders"
>>>                                                                                                         in
>>>                                                                                                         a
>>>                                                                                                         normal
>>>                                                                                                         SQL
>>>                                                                                                         statement.
>>>                                                                                                         This
>>>                                                                                                         is
>>>                                                                                                         because
>>>                                                                                                         without
>>>                                                                                                         a
>>>                                                                                                         "default-catalog",
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         name
>>>                                                                                                         of
>>>                                                                                                         "sales.orders"
>>>                                                                                                         would
>>>                                                                                                         default
>>>                                                                                                         to
>>>                                                                                                         "catalogA".
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         However,
>>>                                                                                                         I
>>>                                                                                                         like
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         design
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         spec,
>>>                                                                                                         because
>>>                                                                                                         it
>>>                                                                                                         has
>>>                                                                                                         the
>>>                                                                                                         "closure"
>>>                                                                                                         property.
>>>                                                                                                         Because
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         fact
>>>                                                                                                         that
>>>                                                                                                         the
>>>                                                                                                         "view
>>>                                                                                                         catalog"
>>>                                                                                                         has
>>>                                                                                                         to
>>>                                                                                                         be
>>>                                                                                                         known
>>>                                                                                                         when
>>>                                                                                                         executing
>>>                                                                                                         a
>>>                                                                                                         view,
>>>                                                                                                         all
>>>                                                                                                         the
>>>                                                                                                         information
>>>                                                                                                         required
>>>                                                                                                         to
>>>                                                                                                         resolve
>>>                                                                                                         the
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         is
>>>                                                                                                         contained
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         metadata
>>>                                                                                                         (and
>>>                                                                                                         the
>>>                                                                                                         "view
>>>                                                                                                         catalog").
>>>                                                                                                         I
>>>                                                                                                         think
>>>                                                                                                         that
>>>                                                                                                         if
>>>                                                                                                         you
>>>                                                                                                         make
>>>                                                                                                         the
>>>                                                                                                         identifier
>>>                                                                                                         resolution
>>>                                                                                                         dependent
>>>                                                                                                         on
>>>                                                                                                         external
>>>                                                                                                         parameters,
>>>                                                                                                         it
>>>                                                                                                         hinders
>>>                                                                                                         portability.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         Jan
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         On
>>>                                                                                                         4/22/25
>>>                                                                                                         18:36,
>>>                                                                                                         Walaa
>>>                                                                                                         Eldin
>>>                                                                                                         Moustafa
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         Hi
>>>                                                                                                         Jan,
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         Thanks
>>>                                                                                                         for
>>>                                                                                                         the
>>>                                                                                                         thoughtful
>>>                                                                                                         feedback.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         I
>>>                                                                                                         think
>>>                                                                                                         it’s
>>>                                                                                                         important
>>>                                                                                                         we
>>>                                                                                                         clarify
>>>                                                                                                         a
>>>                                                                                                         key
>>>                                                                                                         point
>>>                                                                                                         before
>>>                                                                                                         going
>>>                                                                                                         deeper:
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         Non-determinism
>>>                                                                                                         is
>>>                                                                                                         not
>>>                                                                                                         caused
>>>                                                                                                         by
>>>                                                                                                         session
>>>                                                                                                         fallback
>>>                                                                                                         behavior—it’s
>>>                                                                                                         a
>>>                                                                                                         fundamental
>>>                                                                                                         limitation
>>>                                                                                                         of
>>>                                                                                                         using
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         alone,
>>>                                                                                                         regardless
>>>                                                                                                         of
>>>                                                                                                         whether
>>>                                                                                                         we
>>>                                                                                                         use
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         rule,
>>>                                                                                                         the
>>>                                                                                                         proposed
>>>                                                                                                         fallback
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         session’s
>>>                                                                                                         default
>>>                                                                                                         catalog,
>>>                                                                                                         or
>>>                                                                                                         even
>>>                                                                                                         early
>>>                                                                                                         vs.
>>>                                                                                                         late
>>>                                                                                                         binding.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         The
>>>                                                                                                         same
>>>                                                                                                         fully
>>>                                                                                                         qualified
>>>                                                                                                         identifier
>>>                                                                                                         (e.g.,
>>>                                                                                                         catalogA.namespace.table)
>>>                                                                                                         can
>>>                                                                                                         resolve
>>>                                                                                                         to
>>>                                                                                                         different
>>>                                                                                                         objects
>>>                                                                                                         depending
>>>                                                                                                         solely
>>>                                                                                                         on
>>>                                                                                                         engine-specific
>>>                                                                                                         routing
>>>                                                                                                         logic
>>>                                                                                                         or
>>>                                                                                                         catalog
>>>                                                                                                         aliases.
>>>                                                                                                         So
>>>                                                                                                         determinism
>>>                                                                                                         isn’t
>>>                                                                                                         guaranteed
>>>                                                                                                         just
>>>                                                                                                         because
>>>                                                                                                         an
>>>                                                                                                         identifier
>>>                                                                                                         is
>>>                                                                                                         "fully
>>>                                                                                                         qualified."
>>>                                                                                                         The
>>>                                                                                                         only
>>>                                                                                                         reliable
>>>                                                                                                         anchor
>>>                                                                                                         for
>>>                                                                                                         identity
>>>                                                                                                         is
>>>                                                                                                         the
>>>                                                                                                         UUID.
>>>                                                                                                         That’s
>>>                                                                                                         why
>>>                                                                                                         the
>>>                                                                                                         proposed
>>>                                                                                                         use
>>>                                                                                                         of
>>>                                                                                                         UUIDs
>>>                                                                                                         is
>>>                                                                                                         not
>>>                                                                                                         just
>>>                                                                                                         a
>>>                                                                                                         hardening
>>>                                                                                                         strategy.
>>>                                                                                                         It’s
>>>                                                                                                         the
>>>                                                                                                         actual
>>>                                                                                                         fix
>>>                                                                                                         for
>>>                                                                                                         correctness.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         To
>>>                                                                                                         move
>>>                                                                                                         the
>>>                                                                                                         conversation
>>>                                                                                                         forward,
>>>                                                                                                         could
>>>                                                                                                         you
>>>                                                                                                         help
>>>                                                                                                         clarify
>>>                                                                                                         two
>>>                                                                                                         things
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         context
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         spec:
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         *
>>>                                                                                                         Where
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         metadata
>>>                                                                                                         is
>>>                                                                                                         the
>>>                                                                                                         “view
>>>                                                                                                         catalog”
>>>                                                                                                         stored,
>>>                                                                                                         so
>>>                                                                                                         that
>>>                                                                                                         an
>>>                                                                                                         engine
>>>                                                                                                         knows
>>>                                                                                                         to
>>>                                                                                                         fall
>>>                                                                                                         back
>>>                                                                                                         to
>>>                                                                                                         it
>>>                                                                                                         if
>>>                                                                                                         default-catalog
>>>                                                                                                         is
>>>                                                                                                         null?
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         *
>>>                                                                                                         Are
>>>                                                                                                         we
>>>                                                                                                         even
>>>                                                                                                         allowed
>>>                                                                                                         to
>>>                                                                                                         create
>>>                                                                                                         views
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         session's
>>>                                                                                                         default
>>>                                                                                                         catalog
>>>                                                                                                         (i.e.,
>>>                                                                                                         without
>>>                                                                                                         specifying
>>>                                                                                                         a
>>>                                                                                                         catalog)
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         Iceberg
>>>                                                                                                         spec?
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         These
>>>                                                                                                         questions
>>>                                                                                                         are
>>>                                                                                                         important
>>>                                                                                                         because
>>>                                                                                                         if
>>>                                                                                                         we
>>>                                                                                                         can’t
>>>                                                                                                         unambiguously
>>>                                                                                                         recover
>>>                                                                                                         the
>>>                                                                                                         "view
>>>                                                                                                         catalog"
>>>                                                                                                         from
>>>                                                                                                         metadata,
>>>                                                                                                         then
>>>                                                                                                         defaulting
>>>                                                                                                         to
>>>                                                                                                         it
>>>                                                                                                         is
>>>                                                                                                         problematic.
>>>                                                                                                         And
>>>                                                                                                         if
>>>                                                                                                         views
>>>                                                                                                         can't
>>>                                                                                                         be
>>>                                                                                                         created
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         default
>>>                                                                                                         catalog,
>>>                                                                                                         then
>>>                                                                                                         the
>>>                                                                                                         fallback
>>>                                                                                                         rule
>>>                                                                                                         doesn’t
>>>                                                                                                         generalize.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>>>>
>>>                                                                                                         Walaa.
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         >>>>>
>>>                                                                                                         On
>>>                                                                                                         Tue,
>>>                                                                                                         Apr
>>>                                                                                                         22,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         3:14 AM
>>>                                                                                                         Jan
>>>                                                                                                         Kaul
>>>                                                                                                         <ja...@mailbox.org.invalid>
>>>                                                                                                         <ma...@mailbox.org.invalid>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Hi
>>>                                                                                                         Walaa,
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         thank
>>>                                                                                                         you
>>>                                                                                                         for
>>>                                                                                                         your
>>>                                                                                                         proposal.
>>>                                                                                                         If
>>>                                                                                                         I
>>>                                                                                                         understood
>>>                                                                                                         correctly,
>>>                                                                                                         you
>>>                                                                                                         proposal
>>>                                                                                                         is
>>>                                                                                                         composed
>>>                                                                                                         of
>>>                                                                                                         three
>>>                                                                                                         parts:
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         -
>>>                                                                                                         session
>>>                                                                                                         default
>>>                                                                                                         catalog
>>>                                                                                                         as
>>>                                                                                                         fallback
>>>                                                                                                         for
>>>                                                                                                         "default-catalog"
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         -
>>>                                                                                                         session
>>>                                                                                                         default
>>>                                                                                                         namespace
>>>                                                                                                         as
>>>                                                                                                         fallback
>>>                                                                                                         for
>>>                                                                                                         "default-namepace"
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         -
>>>                                                                                                         Late
>>>                                                                                                         binding
>>>                                                                                                         +
>>>                                                                                                         UUID
>>>                                                                                                         validation
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         I
>>>                                                                                                         have
>>>                                                                                                         some
>>>                                                                                                         comments
>>>                                                                                                         regarding
>>>                                                                                                         these
>>>                                                                                                         points.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         1.
>>>                                                                                                         Session
>>>                                                                                                         default
>>>                                                                                                         catalog
>>>                                                                                                         as
>>>                                                                                                         fallback
>>>                                                                                                         for
>>>                                                                                                         "default-catalog"
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Introducing
>>>                                                                                                         a
>>>                                                                                                         behavior
>>>                                                                                                         that
>>>                                                                                                         depends
>>>                                                                                                         on
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         session
>>>                                                                                                         setup
>>>                                                                                                         is
>>>                                                                                                         in
>>>                                                                                                         my
>>>                                                                                                         opinion
>>>                                                                                                         the
>>>                                                                                                         definition
>>>                                                                                                         of
>>>                                                                                                         "non-determinism".
>>>                                                                                                         You
>>>                                                                                                         could
>>>                                                                                                         be
>>>                                                                                                         running
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         query-engine
>>>                                                                                                         and
>>>                                                                                                         catalog-setup
>>>                                                                                                         on
>>>                                                                                                         different
>>>                                                                                                         days,
>>>                                                                                                         with
>>>                                                                                                         different
>>>                                                                                                         default
>>>                                                                                                         session
>>>                                                                                                         catalogs
>>>                                                                                                         (which
>>>                                                                                                         is
>>>                                                                                                         rather
>>>                                                                                                         common),
>>>                                                                                                         and
>>>                                                                                                         would
>>>                                                                                                         be
>>>                                                                                                         getting
>>>                                                                                                         different
>>>                                                                                                         results.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Whereas
>>>                                                                                                         with
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         behavior,
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         always
>>>                                                                                                         produces
>>>                                                                                                         the
>>>                                                                                                         same
>>>                                                                                                         results.
>>>                                                                                                         The
>>>                                                                                                         current
>>>                                                                                                         behavior
>>>                                                                                                         has
>>>                                                                                                         some
>>>                                                                                                         rough
>>>                                                                                                         edges
>>>                                                                                                         in
>>>                                                                                                         very
>>>                                                                                                         niche
>>>                                                                                                         use
>>>                                                                                                         cases
>>>                                                                                                         but
>>>                                                                                                         I
>>>                                                                                                         think
>>>                                                                                                         is
>>>                                                                                                         solid
>>>                                                                                                         for
>>>                                                                                                         most
>>>                                                                                                         uses
>>>                                                                                                         cases.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         2.
>>>                                                                                                         Session
>>>                                                                                                         default
>>>                                                                                                         namespace
>>>                                                                                                         as
>>>                                                                                                         fallback
>>>                                                                                                         for
>>>                                                                                                         "default-namespace"
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Similar
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         above.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         3.
>>>                                                                                                         Late
>>>                                                                                                         binding
>>>                                                                                                         +
>>>                                                                                                         UUID
>>>                                                                                                         validation
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         If
>>>                                                                                                         I
>>>                                                                                                         understand
>>>                                                                                                         it
>>>                                                                                                         correctly,
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         implementation
>>>                                                                                                         already
>>>                                                                                                         uses
>>>                                                                                                         late
>>>                                                                                                         binding.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Generally,
>>>                                                                                                         having
>>>                                                                                                         UUID
>>>                                                                                                         validation
>>>                                                                                                         makes
>>>                                                                                                         the
>>>                                                                                                         setup
>>>                                                                                                         more
>>>                                                                                                         robust.
>>>                                                                                                         Which
>>>                                                                                                         is
>>>                                                                                                         great.
>>>                                                                                                         However,
>>>                                                                                                         having
>>>                                                                                                         UUID
>>>                                                                                                         validation
>>>                                                                                                         still
>>>                                                                                                         requires
>>>                                                                                                         us
>>>                                                                                                         to
>>>                                                                                                         have
>>>                                                                                                         a
>>>                                                                                                         portable
>>>                                                                                                         table
>>>                                                                                                         identifier
>>>                                                                                                         specification.
>>>                                                                                                         Even
>>>                                                                                                         if
>>>                                                                                                         we
>>>                                                                                                         have
>>>                                                                                                         the
>>>                                                                                                         UUIDs
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         referenced
>>>                                                                                                         tables
>>>                                                                                                         from
>>>                                                                                                         the
>>>                                                                                                         view,
>>>                                                                                                         there
>>>                                                                                                         simply
>>>                                                                                                         isn't
>>>                                                                                                         an
>>>                                                                                                         interface
>>>                                                                                                         that
>>>                                                                                                         let's
>>>                                                                                                         us
>>>                                                                                                         use
>>>                                                                                                         those
>>>                                                                                                         UUIDs.
>>>                                                                                                         The
>>>                                                                                                         catalog
>>>                                                                                                         interface
>>>                                                                                                         is
>>>                                                                                                         defined
>>>                                                                                                         in
>>>                                                                                                         terms
>>>                                                                                                         of
>>>                                                                                                         table
>>>                                                                                                         identifiers.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         So
>>>                                                                                                         we
>>>                                                                                                         always
>>>                                                                                                         require
>>>                                                                                                         a
>>>                                                                                                         working
>>>                                                                                                         catalog
>>>                                                                                                         setup
>>>                                                                                                         and
>>>                                                                                                         suiting
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         to
>>>                                                                                                         obtain
>>>                                                                                                         the
>>>                                                                                                         table
>>>                                                                                                         metadata.
>>>                                                                                                         We
>>>                                                                                                         can
>>>                                                                                                         use
>>>                                                                                                         the
>>>                                                                                                         UUIDs
>>>                                                                                                         to
>>>                                                                                                         verify
>>>                                                                                                         if
>>>                                                                                                         we
>>>                                                                                                         loaded
>>>                                                                                                         the
>>>                                                                                                         correct
>>>                                                                                                         table.
>>>                                                                                                         But
>>>                                                                                                         this
>>>                                                                                                         can
>>>                                                                                                         only
>>>                                                                                                         be
>>>                                                                                                         done
>>>                                                                                                         after
>>>                                                                                                         we
>>>                                                                                                         used
>>>                                                                                                         some
>>>                                                                                                         identifier.
>>>                                                                                                         Which
>>>                                                                                                         means
>>>                                                                                                         there
>>>                                                                                                         is
>>>                                                                                                         no
>>>                                                                                                         way
>>>                                                                                                         of
>>>                                                                                                         using
>>>                                                                                                         UUIDs
>>>                                                                                                         without
>>>                                                                                                         a
>>>                                                                                                         functioning
>>>                                                                                                         catalog/identifier
>>>                                                                                                         setup.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         In
>>>                                                                                                         conclusion,
>>>                                                                                                         I
>>>                                                                                                         prefer
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         behavior
>>>                                                                                                         for
>>>                                                                                                         "default-catalog"
>>>                                                                                                         because
>>>                                                                                                         it
>>>                                                                                                         is
>>>                                                                                                         more
>>>                                                                                                         deterministic
>>>                                                                                                         in
>>>                                                                                                         my
>>>                                                                                                         opinion.
>>>                                                                                                         And
>>>                                                                                                         I
>>>                                                                                                         think
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         spec
>>>                                                                                                         does
>>>                                                                                                         a
>>>                                                                                                         good
>>>                                                                                                         job
>>>                                                                                                         for
>>>                                                                                                         multi-engine
>>>                                                                                                         table
>>>                                                                                                         identifier
>>>                                                                                                         resolution.
>>>                                                                                                         I
>>>                                                                                                         see
>>>                                                                                                         the
>>>                                                                                                         UUID
>>>                                                                                                         validation
>>>                                                                                                         more
>>>                                                                                                         of
>>>                                                                                                         an
>>>                                                                                                         additional
>>>                                                                                                         hardening
>>>                                                                                                         strategy.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Thanks
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Jan
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         On
>>>                                                                                                         4/21/25
>>>                                                                                                         17:38,
>>>                                                                                                         Walaa
>>>                                                                                                         Eldin
>>>                                                                                                         Moustafa
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Thanks
>>>                                                                                                         Renjie!
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         The
>>>                                                                                                         existing
>>>                                                                                                         spec
>>>                                                                                                         has
>>>                                                                                                         some
>>>                                                                                                         guidance
>>>                                                                                                         on
>>>                                                                                                         resolving
>>>                                                                                                         catalogs
>>>                                                                                                         on
>>>                                                                                                         the
>>>                                                                                                         fly
>>>                                                                                                         already
>>>                                                                                                         (to
>>>                                                                                                         address
>>>                                                                                                         the
>>>                                                                                                         case
>>>                                                                                                         of
>>>                                                                                                         view
>>>                                                                                                         text
>>>                                                                                                         with
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         missing
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         part).
>>>                                                                                                         The
>>>                                                                                                         guidance
>>>                                                                                                         is
>>>                                                                                                         to
>>>                                                                                                         use
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         where
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         stored.
>>>                                                                                                         But
>>>                                                                                                         I
>>>                                                                                                         find
>>>                                                                                                         this
>>>                                                                                                         rule
>>>                                                                                                         hard
>>>                                                                                                         to
>>>                                                                                                         interpret
>>>                                                                                                         or
>>>                                                                                                         use.
>>>                                                                                                         The
>>>                                                                                                         catalog
>>>                                                                                                         itself
>>>                                                                                                         is
>>>                                                                                                         a
>>>                                                                                                         logical
>>>                                                                                                         construct—such
>>>                                                                                                         as
>>>                                                                                                         a
>>>                                                                                                         federated
>>>                                                                                                         catalog
>>>                                                                                                         that
>>>                                                                                                         delegates
>>>                                                                                                         to
>>>                                                                                                         multiple
>>>                                                                                                         physical
>>>                                                                                                         backends
>>>                                                                                                         (e.g.,
>>>                                                                                                         HMS
>>>                                                                                                         and
>>>                                                                                                         REST).
>>>                                                                                                         In
>>>                                                                                                         such
>>>                                                                                                         cases,
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         (e.g.,
>>>                                                                                                         `my_catalog`
>>>                                                                                                         in
>>>                                                                                                         `my_catalog.namespace1.table1`)
>>>                                                                                                         doesn’t
>>>                                                                                                         physically
>>>                                                                                                         store
>>>                                                                                                         the
>>>                                                                                                         tables;
>>>                                                                                                         it
>>>                                                                                                         only
>>>                                                                                                         routes
>>>                                                                                                         requests
>>>                                                                                                         to
>>>                                                                                                         underlying
>>>                                                                                                         stores.
>>>                                                                                                         Therefore,
>>>                                                                                                         defaulting
>>>                                                                                                         identifier
>>>                                                                                                         resolution
>>>                                                                                                         based
>>>                                                                                                         on
>>>                                                                                                         the
>>>                                                                                                         catalog
>>>                                                                                                         where
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         is
>>>                                                                                                         "stored"
>>>                                                                                                         doesn’t
>>>                                                                                                         align
>>>                                                                                                         with
>>>                                                                                                         how
>>>                                                                                                         catalogs
>>>                                                                                                         actually
>>>                                                                                                         behave
>>>                                                                                                         in
>>>                                                                                                         practice.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>>>>>
>>>                                                                                                         Walaa.
>>>                                                                                                         >>>>>>
>>>                                                                                                         >>>>>>
>>>                                                                                                         On
>>>                                                                                                         Sun,
>>>                                                                                                         Apr
>>>                                                                                                         20,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         11:17 PM
>>>                                                                                                         Renjie
>>>                                                                                                         Liu
>>>                                                                                                         <li...@gmail.com>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>>>
>>>                                                                                                         >>>>>>>
>>>                                                                                                         Hi,
>>>                                                                                                         Walaa:
>>>                                                                                                         >>>>>>>
>>>                                                                                                         >>>>>>>
>>>                                                                                                         Thanks
>>>                                                                                                         for
>>>                                                                                                         the
>>>                                                                                                         proposal.
>>>                                                                                                         >>>>>>>
>>>                                                                                                         >>>>>>>
>>>                                                                                                         I've
>>>                                                                                                         reviewed
>>>                                                                                                         the
>>>                                                                                                         doc,
>>>                                                                                                         but
>>>                                                                                                         in
>>>                                                                                                         general
>>>                                                                                                         I
>>>                                                                                                         have
>>>                                                                                                         some
>>>                                                                                                         concerns
>>>                                                                                                         with
>>>                                                                                                         resolving
>>>                                                                                                         catalog
>>>                                                                                                         names
>>>                                                                                                         on
>>>                                                                                                         the
>>>                                                                                                         fly
>>>                                                                                                         with
>>>                                                                                                         query
>>>                                                                                                         engine
>>>                                                                                                         defined
>>>                                                                                                         catalog
>>>                                                                                                         names.
>>>                                                                                                         This
>>>                                                                                                         introduces
>>>                                                                                                         some
>>>                                                                                                         flexibility
>>>                                                                                                         at
>>>                                                                                                         first
>>>                                                                                                         glance,
>>>                                                                                                         but
>>>                                                                                                         also
>>>                                                                                                         makes
>>>                                                                                                         misconfiguration
>>>                                                                                                         difficult
>>>                                                                                                         to
>>>                                                                                                         explain.
>>>                                                                                                         >>>>>>>
>>>                                                                                                         >>>>>>>
>>>                                                                                                         But
>>>                                                                                                         I
>>>                                                                                                         agree
>>>                                                                                                         with
>>>                                                                                                         one
>>>                                                                                                         part
>>>                                                                                                         that
>>>                                                                                                         we
>>>                                                                                                         should
>>>                                                                                                         store
>>>                                                                                                         resolved
>>>                                                                                                         table
>>>                                                                                                         uuid
>>>                                                                                                         in
>>>                                                                                                         view
>>>                                                                                                         metadata,
>>>                                                                                                         as
>>>                                                                                                         table/view
>>>                                                                                                         renaming
>>>                                                                                                         may
>>>                                                                                                         introduce
>>>                                                                                                         errors
>>>                                                                                                         that's
>>>                                                                                                         difficult
>>>                                                                                                         to
>>>                                                                                                         understand
>>>                                                                                                         for
>>>                                                                                                         user.
>>>                                                                                                         >>>>>>>
>>>                                                                                                         >>>>>>>
>>>                                                                                                         On
>>>                                                                                                         Sat,
>>>                                                                                                         Apr
>>>                                                                                                         19,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         3:02 AM
>>>                                                                                                         Walaa
>>>                                                                                                         Eldin
>>>                                                                                                         Moustafa
>>>                                                                                                         <wa...@gmail.com>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         Hi
>>>                                                                                                         Everyone,
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         Looking
>>>                                                                                                         forward
>>>                                                                                                         to
>>>                                                                                                         keeping
>>>                                                                                                         up
>>>                                                                                                         the
>>>                                                                                                         momentum
>>>                                                                                                         and
>>>                                                                                                         closing
>>>                                                                                                         out
>>>                                                                                                         the
>>>                                                                                                         MV
>>>                                                                                                         spec
>>>                                                                                                         as
>>>                                                                                                         well.
>>>                                                                                                         I’m
>>>                                                                                                         hoping
>>>                                                                                                         we
>>>                                                                                                         can
>>>                                                                                                         proceed
>>>                                                                                                         to
>>>                                                                                                         a
>>>                                                                                                         vote
>>>                                                                                                         next
>>>                                                                                                         week.
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         Here
>>>                                                                                                         is
>>>                                                                                                         a
>>>                                                                                                         summary
>>>                                                                                                         in
>>>                                                                                                         case
>>>                                                                                                         that
>>>                                                                                                         helps.
>>>                                                                                                         The
>>>                                                                                                         proposal
>>>                                                                                                         outlines
>>>                                                                                                         a
>>>                                                                                                         strategy
>>>                                                                                                         for
>>>                                                                                                         handling
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         in
>>>                                                                                                         Iceberg
>>>                                                                                                         view
>>>                                                                                                         metadata,
>>>                                                                                                         with
>>>                                                                                                         the
>>>                                                                                                         goal
>>>                                                                                                         of
>>>                                                                                                         ensuring
>>>                                                                                                         correctness,
>>>                                                                                                         portability,
>>>                                                                                                         and
>>>                                                                                                         engine
>>>                                                                                                         compatibility.
>>>                                                                                                         It
>>>                                                                                                         recommends
>>>                                                                                                         resolving
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         at
>>>                                                                                                         read
>>>                                                                                                         time
>>>                                                                                                         (late
>>>                                                                                                         binding)
>>>                                                                                                         rather
>>>                                                                                                         than
>>>                                                                                                         creation
>>>                                                                                                         time,
>>>                                                                                                         and
>>>                                                                                                         introduces
>>>                                                                                                         UUID-based
>>>                                                                                                         validation
>>>                                                                                                         to
>>>                                                                                                         maintain
>>>                                                                                                         identity
>>>                                                                                                         guarantees
>>>                                                                                                         across
>>>                                                                                                         engines,
>>>                                                                                                         or
>>>                                                                                                         sessions.
>>>                                                                                                         It
>>>                                                                                                         also
>>>                                                                                                         revises
>>>                                                                                                         how
>>>                                                                                                         default-catalog
>>>                                                                                                         and
>>>                                                                                                         default-namespace
>>>                                                                                                         are
>>>                                                                                                         handled
>>>                                                                                                         (defaulting
>>>                                                                                                         both
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         session
>>>                                                                                                         context
>>>                                                                                                         if
>>>                                                                                                         not
>>>                                                                                                         explicitly
>>>                                                                                                         set)
>>>                                                                                                         to
>>>                                                                                                         better
>>>                                                                                                         align
>>>                                                                                                         with
>>>                                                                                                         engine
>>>                                                                                                         behavior
>>>                                                                                                         and
>>>                                                                                                         improve
>>>                                                                                                         cross-engine
>>>                                                                                                         interoperability.
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         Please
>>>                                                                                                         let
>>>                                                                                                         me
>>>                                                                                                         know
>>>                                                                                                         your
>>>                                                                                                         thoughts.
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         Walaa.
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         >>>>>>>>
>>>                                                                                                         On
>>>                                                                                                         Wed,
>>>                                                                                                         Apr
>>>                                                                                                         16,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         2:03 PM
>>>                                                                                                         Walaa
>>>                                                                                                         Eldin
>>>                                                                                                         Moustafa
>>>                                                                                                         <wa...@gmail.com>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         Thanks
>>>                                                                                                         Eduard
>>>                                                                                                         and
>>>                                                                                                         Sung!
>>>                                                                                                         I
>>>                                                                                                         have
>>>                                                                                                         addressed
>>>                                                                                                         the
>>>                                                                                                         comments.
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         One
>>>                                                                                                         key
>>>                                                                                                         point
>>>                                                                                                         to
>>>                                                                                                         keep
>>>                                                                                                         in
>>>                                                                                                         mind
>>>                                                                                                         is
>>>                                                                                                         that
>>>                                                                                                         catalog
>>>                                                                                                         names
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         spec
>>>                                                                                                         refer
>>>                                                                                                         to
>>>                                                                                                         logical
>>>                                                                                                         catalogs—i.e.,
>>>                                                                                                         the
>>>                                                                                                         first
>>>                                                                                                         part
>>>                                                                                                         of
>>>                                                                                                         a
>>>                                                                                                         three-part
>>>                                                                                                         identifier.
>>>                                                                                                         These
>>>                                                                                                         correspond
>>>                                                                                                         to
>>>                                                                                                         Spark's
>>>                                                                                                         DataSourceV2
>>>                                                                                                         catalogs,
>>>                                                                                                         Trino
>>>                                                                                                         connectors,
>>>                                                                                                         and
>>>                                                                                                         similar
>>>                                                                                                         constructs.
>>>                                                                                                         This
>>>                                                                                                         is
>>>                                                                                                         a
>>>                                                                                                         level
>>>                                                                                                         of
>>>                                                                                                         abstraction
>>>                                                                                                         above
>>>                                                                                                         physical
>>>                                                                                                         catalogs,
>>>                                                                                                         which
>>>                                                                                                         are
>>>                                                                                                         not
>>>                                                                                                         referenced
>>>                                                                                                         or
>>>                                                                                                         used
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         spec.
>>>                                                                                                         The
>>>                                                                                                         reason
>>>                                                                                                         is
>>>                                                                                                         that
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         definition/text
>>>                                                                                                         itself
>>>                                                                                                         refer
>>>                                                                                                         to
>>>                                                                                                         logical
>>>                                                                                                         catalogs,
>>>                                                                                                         not
>>>                                                                                                         physical
>>>                                                                                                         ones
>>>                                                                                                         (since
>>>                                                                                                         they
>>>                                                                                                         interface
>>>                                                                                                         directly
>>>                                                                                                         with
>>>                                                                                                         the
>>>                                                                                                         engine
>>>                                                                                                         and
>>>                                                                                                         not
>>>                                                                                                         a
>>>                                                                                                         specific
>>>                                                                                                         metastore).
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         Walaa.
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         >>>>>>>>>
>>>                                                                                                         On
>>>                                                                                                         Wed,
>>>                                                                                                         Apr
>>>                                                                                                         16,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         6:15 AM
>>>                                                                                                         Sung
>>>                                                                                                         Yun
>>>                                                                                                         <su...@gmail.com>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>
>>>                                                                                                         Thank
>>>                                                                                                         you
>>>                                                                                                         Walaa
>>>                                                                                                         for
>>>                                                                                                         the
>>>                                                                                                         proposal.
>>>                                                                                                         I
>>>                                                                                                         think
>>>                                                                                                         view
>>>                                                                                                         portability
>>>                                                                                                         is
>>>                                                                                                         a
>>>                                                                                                         very
>>>                                                                                                         important
>>>                                                                                                         topic
>>>                                                                                                         for
>>>                                                                                                         us
>>>                                                                                                         to
>>>                                                                                                         continue
>>>                                                                                                         discussing
>>>                                                                                                         as
>>>                                                                                                         it
>>>                                                                                                         relies
>>>                                                                                                         on
>>>                                                                                                         many
>>>                                                                                                         assumptions
>>>                                                                                                         within
>>>                                                                                                         the
>>>                                                                                                         data
>>>                                                                                                         ecosystem
>>>                                                                                                         for
>>>                                                                                                         it
>>>                                                                                                         to
>>>                                                                                                         function
>>>                                                                                                         like
>>>                                                                                                         you've
>>>                                                                                                         highlighted
>>>                                                                                                         well
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         document.
>>>                                                                                                         >>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>
>>>                                                                                                         I've
>>>                                                                                                         added
>>>                                                                                                         a
>>>                                                                                                         few
>>>                                                                                                         comments
>>>                                                                                                         around
>>>                                                                                                         how
>>>                                                                                                         this
>>>                                                                                                         may
>>>                                                                                                         impact
>>>                                                                                                         the
>>>                                                                                                         permission
>>>                                                                                                         questions
>>>                                                                                                         the
>>>                                                                                                         engines
>>>                                                                                                         will
>>>                                                                                                         be
>>>                                                                                                         asking,
>>>                                                                                                         and
>>>                                                                                                         whether
>>>                                                                                                         that
>>>                                                                                                         is
>>>                                                                                                         the
>>>                                                                                                         desired
>>>                                                                                                         behavior.
>>>                                                                                                         >>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>
>>>                                                                                                         Sung
>>>                                                                                                         >>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>
>>>                                                                                                         On
>>>                                                                                                         Wed,
>>>                                                                                                         Apr
>>>                                                                                                         16,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         7:32 AM
>>>                                                                                                         Eduard
>>>                                                                                                         Tudenhöfner
>>>                                                                                                         <et...@apache.org>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>
>>>                                                                                                         Thanks
>>>                                                                                                         Walaa
>>>                                                                                                         for
>>>                                                                                                         tackling
>>>                                                                                                         this
>>>                                                                                                         problem.
>>>                                                                                                         I've
>>>                                                                                                         added
>>>                                                                                                         a
>>>                                                                                                         few
>>>                                                                                                         comments
>>>                                                                                                         to
>>>                                                                                                         get
>>>                                                                                                         a
>>>                                                                                                         better
>>>                                                                                                         understanding
>>>                                                                                                         of
>>>                                                                                                         how
>>>                                                                                                         this
>>>                                                                                                         will
>>>                                                                                                         look
>>>                                                                                                         like
>>>                                                                                                         in
>>>                                                                                                         the
>>>                                                                                                         actual
>>>                                                                                                         implementation.
>>>                                                                                                         >>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>
>>>                                                                                                         Eduard
>>>                                                                                                         >>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>
>>>                                                                                                         On
>>>                                                                                                         Tue,
>>>                                                                                                         Apr
>>>                                                                                                         15,
>>>                                                                                                         2025
>>>                                                                                                         at
>>>                                                                                                         7:09 PM
>>>                                                                                                         Walaa
>>>                                                                                                         Eldin
>>>                                                                                                         Moustafa
>>>                                                                                                         <wa...@gmail.com>
>>>                                                                                                         wrote:
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         Hi
>>>                                                                                                         Everyone,
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         Starting
>>>                                                                                                         this
>>>                                                                                                         thread
>>>                                                                                                         to
>>>                                                                                                         resume
>>>                                                                                                         our
>>>                                                                                                         discussion
>>>                                                                                                         on
>>>                                                                                                         how
>>>                                                                                                         to
>>>                                                                                                         reference
>>>                                                                                                         table
>>>                                                                                                         identifiers
>>>                                                                                                         from
>>>                                                                                                         Iceberg
>>>                                                                                                         metadata,
>>>                                                                                                         a
>>>                                                                                                         key
>>>                                                                                                         aspect
>>>                                                                                                         of
>>>                                                                                                         the
>>>                                                                                                         view
>>>                                                                                                         specification,
>>>                                                                                                         particularly
>>>                                                                                                         in
>>>                                                                                                         relation
>>>                                                                                                         to
>>>                                                                                                         the
>>>                                                                                                         MV
>>>                                                                                                         (materialized
>>>                                                                                                         view)
>>>                                                                                                         extensions.
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         I
>>>                                                                                                         had
>>>                                                                                                         the
>>>                                                                                                         chance
>>>                                                                                                         to
>>>                                                                                                         speak
>>>                                                                                                         offline
>>>                                                                                                         with
>>>                                                                                                         a
>>>                                                                                                         few
>>>                                                                                                         community
>>>                                                                                                         members
>>>                                                                                                         to
>>>                                                                                                         better
>>>                                                                                                         understand
>>>                                                                                                         how
>>>                                                                                                         the
>>>                                                                                                         current
>>>                                                                                                         spec
>>>                                                                                                         is
>>>                                                                                                         being
>>>                                                                                                         interpreted.
>>>                                                                                                         Those
>>>                                                                                                         conversations
>>>                                                                                                         served
>>>                                                                                                         as
>>>                                                                                                         inputs
>>>                                                                                                         to
>>>                                                                                                         a
>>>                                                                                                         new
>>>                                                                                                         proposal
>>>                                                                                                         on
>>>                                                                                                         how
>>>                                                                                                         table
>>>                                                                                                         identifier
>>>                                                                                                         references
>>>                                                                                                         could
>>>                                                                                                         be
>>>                                                                                                         represented
>>>                                                                                                         in
>>>                                                                                                         metadata.
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         You
>>>                                                                                                         can
>>>                                                                                                         find
>>>                                                                                                         the
>>>                                                                                                         proposal
>>>                                                                                                         here
>>>                                                                                                         [1].
>>>                                                                                                         I
>>>                                                                                                         look
>>>                                                                                                         forward
>>>                                                                                                         to
>>>                                                                                                         your
>>>                                                                                                         feedback
>>>                                                                                                         and
>>>                                                                                                         working
>>>                                                                                                         together
>>>                                                                                                         to
>>>                                                                                                         move
>>>                                                                                                         this
>>>                                                                                                         forward
>>>                                                                                                         so
>>>                                                                                                         we
>>>                                                                                                         can
>>>                                                                                                         finalize
>>>                                                                                                         the
>>>                                                                                                         MV
>>>                                                                                                         spec
>>>                                                                                                         as
>>>                                                                                                         well.
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         [1]
>>>                                                                                                         https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         Thanks,
>>>                                                                                                         >>>>>>>>>>>>
>>>                                                                                                         Walaa.
>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Steven Wu <st...@gmail.com>]

Regarding the following representation for the storage table identifier:

* catalog name: optional
* namespace: optional
* table name: required
* UUID: required
It is the same discussion as this github thread.
https://github.com/apache/iceberg/pull/11041/files#r1869998317

I was in favor of adding them mostly from human readability when manually
inspecting the storage table to avoid a reverse lookup from table UUID to
table identifier.  Benny (and a few others) have expressed that it may add
more ambiguity (like this thread was discussing). To clarify, we only need
to include the identifier as they are defined in the SQL query (e.g.
catalog and namespace are optional).  I doubt it would add more ambiguity.
But they are certainly not functionally required. So I am not opinionated
on this one.

On Mon, May 12, 2025 at 5:01 AM Jan Kaul <ja...@mailbox.org.invalid>
wrote:

> I have a question regarding the following representation for the storage
> table identifier:
>
> * catalog name: optional
> * namespace: optional
> * table name: required
> * UUID: required
>
> Having optional fields as part of the identifier again raises the question
> of what to use if these optional fields are not set. And we again have the
> early vs late binding discussion. Where early binding would default to the
> view catalog and late binding would use the current session
> catalog/namespace.
>
> @Walaa: Which behavior did you have in mind?
>
> In my opinion, having the functionality of MVs depend on the current
> session context is not desirable. So I would argue against late binding.
>
> Best wishes,
>
> Jan
> On 5/8/25 19:59, Walaa Eldin Moustafa wrote:
>
> Let us continue here for the context and continuity of the conversation.
>
> On Thu, May 8, 2025 at 10:55 AM Steven Wu <st...@gmail.com> wrote:
>
>> For the storage table identifier discussion, can we move to Jan's PR on
>> materialized view spec?
>> https://github.com/apache/iceberg/pull/11041
>>
>>
>> On Thu, May 8, 2025 at 10:40 AM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> It looks like we may not reach alignment on the resolution method in the
>>> short term. That said, perhaps we can move forward by aligning on the
>>> format used to store table identifiers in metadata instead.
>>>
>>> The idea is that the same format could work under either resolution
>>> model, which would allow us to make progress independently of that debate.
>>>
>>> Is this storage format of the storage table identifier agreeable to you?
>>> * catalog name: optional
>>> * namespace: optional
>>> * table name: required
>>> * UUID: required
>>> With the condition that the components included in a table identifier in
>>> the metadata should match those used in the view text for that table.
>>>
>>> The same structure above could apply to storing child tables or view
>>> identifiers as part of lineage, if and when that’s needed.
>>>
>>> For the longer term resolution model, it might make sense to initiate a
>>> cross-community discussion.
>>>
>>> Thanks,
>>> Walaa.
>>>
>>> On Thu, May 8, 2025 at 9:07 AM Steven Wu <st...@gmail.com> wrote:
>>>
>>>> If a table referenced by the view SQL query is deleted and recreated,
>>>> is the regular view still valid? If yes, it won't be correct to use
>>>> resolved UUID to validate the name reference. On the materialized view
>>>> side, we are aligned that UUID is necessary to detect this scenario and
>>>> mark the refresh state "stale" if a referenced table is re-created.
>>>>
>>>> > esolution using default-catalog and default-namespace does not
>>>> provide full determinism,
>>>>
>>>> With the new proposal, the catalog and namespace depends on the
>>>> engine's current context. if the user changes the current context (catalog
>>>> and database) via SET/USE , the view SQL query may resolve to a different
>>>> table. That is probably a bigger problem.
>>>>
>>>> With the current view spec, the intention is set in the view creation
>>>> time. I would rather have the view producer set the explicit intention
>>>> (than the implicit dependency on the engine context at view consumption
>>>> phase).
>>>>
>>>> Also the view SQL can still reference a table with a catalog name
>>>> component.  Hence the catalog naming still matters.The new proposal won't
>>>> really solve the naming problem.
>>>>
>>>> On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> Hi Dan,
>>>>>
>>>>> Thanks for the clarification! I agree that UUIDs should not be stored
>>>>> in the SQL definition of the view.
>>>>>
>>>>> Just to clarify: the proposal isn't about modifying the view
>>>>> definition itself, but rather about how table identifiers are represented
>>>>> in Iceberg view metadata, which naturally depends on how those identifiers
>>>>> are resolved.
>>>>>
>>>>> For example, in order to store the "storage table identifier" in the
>>>>> metadata:
>>>>> * Should we apply the same resolution rules based on default-catalog
>>>>> and default-namespace?
>>>>> * And if that resolution method is considered deterministic enough,
>>>>> would you be comfortable not storing the UUID in the storage table
>>>>> identifier metadata?
>>>>>
>>>>> The same questions apply when storing child views or tables as part of
>>>>> the view’s lineage metadata. It would be helpful to clarify the resolution
>>>>> model and whether UUIDs should be part of that for validation or drift
>>>>> detection.
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org> wrote:
>>>>>
>>>>>> I don't think we want to include the resolved table UUIDs in the view
>>>>>> definition, but rather in the storage table state.  You can still resolve
>>>>>> whether those drift at some point, but I don't feel like it's a good idea
>>>>>> to capture data in the view that we may allow to drift if there isn't any
>>>>>> requirement that they match.  This also aligns with the identifier
>>>>>> resolution being late binding.
>>>>>>
>>>>>> -Dan
>>>>>>
>>>>>> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Steven! So would you agree that resolution using
>>>>>>> default-catalog and default-namespace does not provide full determinism,
>>>>>>> and requires a supporting safety mechanism?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Walaa.
>>>>>>>
>>>>>>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> > If the current model is considered deterministic, do you think
>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>>
>>>>>>>> I am fine with storing UUIDs for table identifiers in the view.
>>>>>>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>>>>>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>>>>>>> fail the query if mismatch.
>>>>>>>>
>>>>>>>> The UUID change doesn't really change the table identifier
>>>>>>>> resolution rule though. It is more of a safety protection.
>>>>>>>>
>>>>>>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Steven,
>>>>>>>>>
>>>>>>>>> Thanks for the reply.
>>>>>>>>>
>>>>>>>>> > I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>>> Iceberg view spec.
>>>>>>>>>
>>>>>>>>> To clarify, I don't believe the proposal is trying to solve
>>>>>>>>> catalog naming. What it’s doing is simply this:
>>>>>>>>>
>>>>>>>>> * Proposing that table names inside views resolve the same way as
>>>>>>>>> they do elsewhere (e.g., queries).
>>>>>>>>> * Adopting a model that is already widely used and supported in
>>>>>>>>> the existing ecosystem, which allows for:
>>>>>>>>>     -- Renaming catalog aliases
>>>>>>>>>     -- Swapping catalog implementations behind consistent names
>>>>>>>>>     -- Having different default catalog names across engines that
>>>>>>>>> still point to the same underlying tables
>>>>>>>>>
>>>>>>>>> These are common patterns in production data lakes. Saying Iceberg
>>>>>>>>> views cannot operate in those environments feels unrealistic. In practice,
>>>>>>>>> it means the spec breaks down in situations that users encounter regularly.
>>>>>>>>>
>>>>>>>>> > The recommendation of using engines’ current catalog and
>>>>>>>>> database can cause context-dependent resolution results.
>>>>>>>>>
>>>>>>>>> * As noted in the doc and earlier replies, fixing a catalog name
>>>>>>>>> doesn’t actually guarantee determinism either. All the failure scenarios
>>>>>>>>> above still apply even when a default-catalog is stored.
>>>>>>>>> * The current spec also allows default-catalog to be null, in
>>>>>>>>> which case it falls back to the view’s catalog, yet that catalog is
>>>>>>>>> determined based on how the view is referenced in the query, which would be
>>>>>>>>> considered non-deterministic based on the same criteria you shared.
>>>>>>>>> * The only true form of determinism here is UUID-based validation,
>>>>>>>>> which protects against silent drift in any resolution model.
>>>>>>>>>
>>>>>>>>> If the current model is considered deterministic, do you think
>>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>>> Or put another way: Would you be comfortable relying solely on
>>>>>>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>>>>>>> table, without UUID validation?
>>>>>>>>>
>>>>>>>>> +1 on involving other communities. I’m happy to help facilitate a
>>>>>>>>> cross-community discussion if we aren’t able to reach a resolution here.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Walaa.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>>>> Iceberg view spec. I am not convinced that the proposed change will make
>>>>>>>>>> the table identifier resolution more clear and portable. The recommendation
>>>>>>>>>> of using engines' current catalog and database can cause context dependent
>>>>>>>>>> resolution results, which seems non-deterministic to me.
>>>>>>>>>>
>>>>>>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>>>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>>>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>>>>>>> more inputs from the broader community on this part.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> In Spark, I believe that the USE commands sets the current
>>>>>>>>>>> catalog and namespace.  This affects both where the view is created and how
>>>>>>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>>>>>>> saving the current catalog and namespace into the view
>>>>>>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> > I think that's the lesser evil compared to Iceberg
>>>>>>>>>>>> specifying how engines should resolve identifiers
>>>>>>>>>>>>
>>>>>>>>>>>> I think this is also similar to the previous point. It is the
>>>>>>>>>>>> other way around. Right now the spec dictates how to resolve (through
>>>>>>>>>>>> employing a view-specific `default-catalog` field). The proposal is
>>>>>>>>>>>> suggesting to get out of this space and let engines handle it similar
>>>>>>>>>>>> to how they handle all identifiers.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Benny, I think this is a misconception or miscommunication.
>>>>>>>>>>>>> The USE command has no impact on the `default-catalog` field. In fact, the
>>>>>>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>>>>>>> is not the case under the current spec.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > there is no SQL construct today to explicitly set
>>>>>>>>>>>>>> default-catalog
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I generally agree with Dan about requiring consistent catalog
>>>>>>>>>>>>>> names.  I think that's the lesser evil compared to Iceberg specifying how
>>>>>>>>>>>>>> engines should resolve identifiers.  Another thing to consider is that
>>>>>>>>>>>>>> identifier resolution can be very expensive at query validation time if
>>>>>>>>>>>>>> identifiers need to be looked up from a bunch of places.  Hopefully, it
>>>>>>>>>>>>>> should be possible to define a view in such a way that identifiers can be
>>>>>>>>>>>>>> resolved on the first try.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>>>>>>> materialized views.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We may be able to converge on this storage format even if we
>>>>>>>>>>>>>>> haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>>>>>>> identifiers in this way.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm supportive of moving forward with consensus on the
>>>>>>>>>>>>>>> identifier storage format. That said, we may continue to run into questions
>>>>>>>>>>>>>>> related to resolution during implementation. For example: Should the
>>>>>>>>>>>>>>> storage table identifier follow the same default-catalog and
>>>>>>>>>>>>>>> default-namespace resolution behavior as other table references?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We can continue having the discussion if the current view
>>>>>>>>>>>>>>>> spec needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Rishabh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Correction of typo: both engines seem to set
>>>>>>>>>>>>>>>>> default-catalog to the view catalog if it is defined, or to null if the
>>>>>>>>>>>>>>>>> view catalog is not defined.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I agree that catalog renaming is an environmental event,
>>>>>>>>>>>>>>>>>> but it's a real one that happens frequently in practice.
>>>>>>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something
>>>>>>>>>>>>>>>>>> as common as catalog renaming feels very restrictive, and could make the
>>>>>>>>>>>>>>>>>> spec less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>>>>>>> I’m sharing this from the perspective of a large data
>>>>>>>>>>>>>>>>>> lake environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Further, it's worth noting that the table spec is
>>>>>>>>>>>>>>>>>> resilient to catalog renaming, but the view spec is not. If we have an
>>>>>>>>>>>>>>>>>> opportunity to make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>>>>>>> Both specifications are deterministic in their
>>>>>>>>>>>>>>>>>> definition, but one is more fragile to environmental changes than the
>>>>>>>>>>>>>>>>>> other. Improving resilience does not sacrifice determinism. It simply makes
>>>>>>>>>>>>>>>>>> views safer and more portable over time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>>>>>>> Today, both engines seem to set default-catalog to null
>>>>>>>>>>>>>>>>>> if the view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>>>>>>> What was the intended thought process behind this
>>>>>>>>>>>>>>>>>> behavior?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > tables inside views remain reachable after a catalog
>>>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This problem stems from the exact
>>>>>>>>>>>>>>>>>>> environmental/configuration issue that we should not be trying to address.
>>>>>>>>>>>>>>>>>>> I don't think we would expect references to survive a catalog rename.
>>>>>>>>>>>>>>>>>>> That's not something covered by the spec and needs to be handled separately
>>>>>>>>>>>>>>>>>>> as a platform-level migration specific to the affected environment.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The identifier resolution logic is clear and
>>>>>>>>>>>>>>>>>>> deterministic.  It should not matter whether an engine resolves and encodes
>>>>>>>>>>>>>>>>>>> the default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but rather
>>>>>>>>>>>>>>>>>>> view behavior when you start altering the environment around it, which
>>>>>>>>>>>>>>>>>>> isn't something we should be trying to define here.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just
>>>>>>>>>>>>>>>>>>>> catalog naming consistency. The behavior around default-catalog itself
>>>>>>>>>>>>>>>>>>>> introduces resolution inconsistencies even when catalog names are
>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside
>>>>>>>>>>>>>>>>>>>> views remain reachable after a catalog rename. But if it is set to a
>>>>>>>>>>>>>>>>>>>> non-null value, table references will break.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * default-catalog causes table references inside views
>>>>>>>>>>>>>>>>>>>> to be early bound (i.e., bound at view creation time, especially when using
>>>>>>>>>>>>>>>>>>>> a non-null value), while table references inside standalone queries are
>>>>>>>>>>>>>>>>>>>> late bound (bound at query time). This creates inconsistencies when
>>>>>>>>>>>>>>>>>>>> resolving the same table name inside and outside views, even within the
>>>>>>>>>>>>>>>>>>>> same job.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from
>>>>>>>>>>>>>>>>>>>> the spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg
>>>>>>>>>>>>>>>>>>>> with that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> How would you suggest approaching the engine-level
>>>>>>>>>>>>>>>>>>>> changes required to support the current default-catalog field?
>>>>>>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities
>>>>>>>>>>>>>>>>>>>> would align around having table resolution behave inconsistently between
>>>>>>>>>>>>>>>>>>>> queries and views, or inconsistency between Iceberg and other types of
>>>>>>>>>>>>>>>>>>>> views?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I would agree with Jan's summary of why
>>>>>>>>>>>>>>>>>>>>> 'default-catalog' was introduced, but I think we need to step back and
>>>>>>>>>>>>>>>>>>>>> align on what we are really attempting to support in the spec.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent
>>>>>>>>>>>>>>>>>>>>> to address the catalog naming/configuration and would argue that we
>>>>>>>>>>>>>>>>>>>>> shouldn't attempt to do that as part of the spec.  Inconsistently named
>>>>>>>>>>>>>>>>>>>>> catalogs are a reality, but we should consider that a
>>>>>>>>>>>>>>>>>>>>> configuration/environmental issue, not something to solve for in the spec.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We should support and advocate for consistency in
>>>>>>>>>>>>>>>>>>>>> catalog naming and define the spec along those lines.  The fact is that
>>>>>>>>>>>>>>>>>>>>> with all of the recent work that's gone into making catalogs pluggable, it
>>>>>>>>>>>>>>>>>>>>> makes more sense to just register catalog configuration with consistent
>>>>>>>>>>>>>>>>>>>>> names (even if you have to duplicate the configuration for supporting
>>>>>>>>>>>>>>>>>>>>> existing readers/writers).  I think it's better to provide a path toward
>>>>>>>>>>>>>>>>>>>>> consistency than to normalize complicated schemes to workaround the issues
>>>>>>>>>>>>>>>>>>>>> caused by environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late
>>>>>>>>>>>>>>>>>>>>> binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was
>>>>>>>>>>>>>>>>>>>>>> that every query engine uses it to store its session default catalog at the
>>>>>>>>>>>>>>>>>>>>>> time of creating the view. This way the view could be reused in another
>>>>>>>>>>>>>>>>>>>>>> session. The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Generally we have different environments we want to
>>>>>>>>>>>>>>>>>>>>>> support with the view spec:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> When the environment supports it, using consistent
>>>>>>>>>>>>>>>>>>>>>> catalog names can have a great benefit for multi-catalog, multi-engine
>>>>>>>>>>>>>>>>>>>>>> setups. With consistent catalog names, using the "default-catalog" field
>>>>>>>>>>>>>>>>>>>>>> works without any issues.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> This can be the case when different query engines
>>>>>>>>>>>>>>>>>>>>>> refer to the same physical catalog by different names. This often happens
>>>>>>>>>>>>>>>>>>>>>> because different query engines use different strategies to setup the
>>>>>>>>>>>>>>>>>>>>>> catalogs. If catalogs have inconsistent naming, using the "default-catalog"
>>>>>>>>>>>>>>>>>>>>>> field does not work because it is not guaranteed that the catalog name can
>>>>>>>>>>>>>>>>>>>>>> be resolved with another engine. Using the "view catalog" as a fallback is
>>>>>>>>>>>>>>>>>>>>>> a better solution for this use case, as it avoids catalog names altogether.
>>>>>>>>>>>>>>>>>>>>>> It is however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields
>>>>>>>>>>>>>>>>>>>>>> from the view spec completely.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate
>>>>>>>>>>>>>>>>>>>>>> view-level default catalogs or namespaces.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly
>>>>>>>>>>>>>>>>>>>>>> as they appear in the view SQL text.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at
>>>>>>>>>>>>>>>>>>>>>> creation, it should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the
>>>>>>>>>>>>>>>>>>>>>>> confusion the fallback creates. Also just to be clear, at this point and
>>>>>>>>>>>>>>>>>>>>>>> after clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I
>>>>>>>>>>>>>>>>>>>>>>>> don't think we should design something that says first use this catalog,
>>>>>>>>>>>>>>>>>>>>>>>> then that catalog.. or that catalog.  For example, resolving identifiers
>>>>>>>>>>>>>>>>>>>>>>>> using default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I
>>>>>>>>>>>>>>>>>>>>>>>> see no value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>>>>>>> - Interoperability between different engine types
>>>>>>>>>>>>>>>>>>>>>>>> is very hard due to dialect issues... so I think we should focus on
>>>>>>>>>>>>>>>>>>>>>>>> supporting different clusters of the same engine type on a shared catalog.
>>>>>>>>>>>>>>>>>>>>>>>> For example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is
>>>>>>>>>>>>>>>>>>>>>>>> along the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier
>>>>>>>>>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the right approach
>>>>>>>>>>>>>>>>>>>>>>>> is really about templating.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers
>>>>>>>>>>>>>>>>>>>>>>>> in a view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the
>>>>>>>>>>>>>>>>>>>>>>>>> inconsistencies I shared regarding many resolution methods not consistently
>>>>>>>>>>>>>>>>>>>>>>>>> being followed in different scenarios (view vs child table resolution,
>>>>>>>>>>>>>>>>>>>>>>>>> query vs view resolution)? Note these occur when the default catalog is set
>>>>>>>>>>>>>>>>>>>>>>>>> to a non-null value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback
>>>>>>>>>>>>>>>>>>>>>>>>>> should be the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really
>>>>>>>>>>>>>>>>>>>>>>>>>> matter what the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog, this is a
>>>>>>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog, catalog names
>>>>>>>>>>>>>>>>>>>>>>>>>> should be included in the reference name already.
>>>>>>>>>>>>>>>>>>>>>>>>>> So no ambiguity
>>>>>>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a
>>>>>>>>>>>>>>>>>>>>>>>>>> separate problem, which
>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can
>>>>>>>>>>>>>>>>>>>>>>>>>> only recommend that
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage
>>>>>>>>>>>>>>>>>>>>>>>>>> for better
>>>>>>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior
>>>>>>>>>>>>>>>>>>>>>>>>>> to engine's session
>>>>>>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than
>>>>>>>>>>>>>>>>>>>>>>>>>> the current fallback
>>>>>>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation
>>>>>>>>>>>>>>>>>>>>>>>>>> issue for not conforming
>>>>>>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I
>>>>>>>>>>>>>>>>>>>>>>>>>> want to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined
>>>>>>>>>>>>>>>>>>>>>>>>>> at view creation time. It would be circular: the view needs to exist before
>>>>>>>>>>>>>>>>>>>>>>>>>> its metadata (and hence default-catalog) can exist. This is visible in
>>>>>>>>>>>>>>>>>>>>>>>>>> Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog
>>>>>>>>>>>>>>>>>>>>>>>>>> setting would require extending SQL syntax and engine APIs to promote it to
>>>>>>>>>>>>>>>>>>>>>>>>>> a first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs
>>>>>>>>>>>>>>>>>>>>>>>>>> from this idea. Spark resolves table identifiers during view creation using
>>>>>>>>>>>>>>>>>>>>>>>>>> the session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of
>>>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" where the view is going to be created in. It can use
>>>>>>>>>>>>>>>>>>>>>>>>>> that catalog to resolve partial table identifiers if "default-catalog" is
>>>>>>>>>>>>>>>>>>>>>>>>>> not set.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior,
>>>>>>>>>>>>>>>>>>>>>>>>>> where partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the
>>>>>>>>>>>>>>>>>>>>>>>>>> tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec.
>>>>>>>>>>>>>>>>>>>>>>>>>> How do we validate that a table exists without a default catalog, or do we
>>>>>>>>>>>>>>>>>>>>>>>>>> always use the current session catalog?
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same
>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. Just to clarify: when I referred to late binding as
>>>>>>>>>>>>>>>>>>>>>>>>>> “similar” to the proposal, I was acknowledging the distinction between
>>>>>>>>>>>>>>>>>>>>>>>>>> view-level and table-level resolution. But as you noted, both follow a late
>>>>>>>>>>>>>>>>>>>>>>>>>> binding model.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to
>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish between the physical catalog and it's reference used in SQL
>>>>>>>>>>>>>>>>>>>>>>>>>> statements. The important part is that the physical catalog of the view and
>>>>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in it's definition stay consistent. You could create
>>>>>>>>>>>>>>>>>>>>>>>>>> a view in a given physical catalog by referring to it as "catalogA", as in
>>>>>>>>>>>>>>>>>>>>>>>>>> your first point. If you then, given a different setup, refer to the same
>>>>>>>>>>>>>>>>>>>>>>>>>> physical catalog as "catalogB" in another session/environment, the behavior
>>>>>>>>>>>>>>>>>>>>>>>>>> should still work.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point.
>>>>>>>>>>>>>>>>>>>>>>>>>> Late binding applies to the view catalog name and by extension to all
>>>>>>>>>>>>>>>>>>>>>>>>>> partial table references when no "default-catalog" is present. Resolving
>>>>>>>>>>>>>>>>>>>>>>>>>> the view catalog name at query time is not opposed to storing the view
>>>>>>>>>>>>>>>>>>>>>>>>>> metadata in a catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what
>>>>>>>>>>>>>>>>>>>>>>>>>> you mean.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being
>>>>>>>>>>>>>>>>>>>>>>>>>> referenced in a SQL statement. That statement contains the information for
>>>>>>>>>>>>>>>>>>>>>>>>>> the query engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as
>>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * FROM catalogB.namespace.view (after renaming catalogA to catalogB,
>>>>>>>>>>>>>>>>>>>>>>>>>> and keeping everything else the same), then catalogB becomes the view’s
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it
>>>>>>>>>>>>>>>>>>>>>>>>>> sounds to me like the catalog is resolved at query time, based on how the
>>>>>>>>>>>>>>>>>>>>>>>>>> view is referenced, not from any stored metadata. That would imply some
>>>>>>>>>>>>>>>>>>>>>>>>>> sort of a late binding behavior (similar to the proposal), as opposed to
>>>>>>>>>>>>>>>>>>>>>>>>>> using some catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current
>>>>>>>>>>>>>>>>>>>>>>>>>> spec: The view is executed when it's being referenced in a SQL statement.
>>>>>>>>>>>>>>>>>>>>>>>>>> That statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog
>>>>>>>>>>>>>>>>>>>>>>>>>> can be referenced at execution time. As long as that's the case I would
>>>>>>>>>>>>>>>>>>>>>>>>>> assume the view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable
>>>>>>>>>>>>>>>>>>>>>>>>>> because the current specification can lead to some unintuitive behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>> For example for the following statement:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders
>>>>>>>>>>>>>>>>>>>>>>>>>> AS SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the
>>>>>>>>>>>>>>>>>>>>>>>>>> view spec, because it has the "closure" property. Because of the fact that
>>>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" has to be known when executing a view, all the
>>>>>>>>>>>>>>>>>>>>>>>>>> information required to resolve the table identifiers is contained in the
>>>>>>>>>>>>>>>>>>>>>>>>>> view metadata (and the "view catalog"). I think that if you make the
>>>>>>>>>>>>>>>>>>>>>>>>>> identifier resolution dependent on external parameters, it hinders
>>>>>>>>>>>>>>>>>>>>>>>>>> portability.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key
>>>>>>>>>>>>>>>>>>>>>>>>>> point before going deeper:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you
>>>>>>>>>>>>>>>>>>>>>>>>>> help clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog” stored, so that an engine knows to fall back to it if
>>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog is null?
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in
>>>>>>>>>>>>>>>>>>>>>>>>>> the session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we
>>>>>>>>>>>>>>>>>>>>>>>>>> can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I
>>>>>>>>>>>>>>>>>>>>>>>>>> understood correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-namepace"
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these
>>>>>>>>>>>>>>>>>>>>>>>>>> points.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the
>>>>>>>>>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of "non-determinism".
>>>>>>>>>>>>>>>>>>>>>>>>>> You could be running the same query-engine and catalog-setup on different
>>>>>>>>>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is rather common), and
>>>>>>>>>>>>>>>>>>>>>>>>>> would be getting different results.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the
>>>>>>>>>>>>>>>>>>>>>>>>>> view always produces the same results. The current behavior has some rough
>>>>>>>>>>>>>>>>>>>>>>>>>> edges in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback
>>>>>>>>>>>>>>>>>>>>>>>>>> for "default-namespace"
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes
>>>>>>>>>>>>>>>>>>>>>>>>>> the setup more robust. Which is great. However, having UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>>> still requires us to have a portable table identifier specification. Even
>>>>>>>>>>>>>>>>>>>>>>>>>> if we have the UUIDs of the referenced tables from the view, there simply
>>>>>>>>>>>>>>>>>>>>>>>>>> isn't an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog
>>>>>>>>>>>>>>>>>>>>>>>>>> setup and suiting table identifiers to obtain the table metadata. We can
>>>>>>>>>>>>>>>>>>>>>>>>>> use the UUIDs to verify if we loaded the correct table. But this can only
>>>>>>>>>>>>>>>>>>>>>>>>>> be done after we used some identifier. Which means there is no way of using
>>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current
>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for "default-catalog" because it is more deterministic in my
>>>>>>>>>>>>>>>>>>>>>>>>>> opinion. And I think the current spec does a good job for multi-engine
>>>>>>>>>>>>>>>>>>>>>>>>>> table identifier resolution. I see the UUID validation more of an
>>>>>>>>>>>>>>>>>>>>>>>>>> additional hardening strategy.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie
>>>>>>>>>>>>>>>>>>>>>>>>>> Liu <li...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I
>>>>>>>>>>>>>>>>>>>>>>>>>> have some concerns with resolving catalog names on the fly with query
>>>>>>>>>>>>>>>>>>>>>>>>>> engine defined catalog names. This introduces some flexibility at first
>>>>>>>>>>>>>>>>>>>>>>>>>> glance, but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should
>>>>>>>>>>>>>>>>>>>>>>>>>> store resolved table uuid in view metadata, as table/view renaming may
>>>>>>>>>>>>>>>>>>>>>>>>>> introduce errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the
>>>>>>>>>>>>>>>>>>>>>>>>>> momentum and closing out the MV spec as well. I’m hoping we can proceed to
>>>>>>>>>>>>>>>>>>>>>>>>>> a vote next week.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps.
>>>>>>>>>>>>>>>>>>>>>>>>>> The proposal outlines a strategy for handling table identifiers in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>> view metadata, with the goal of ensuring correctness, portability, and
>>>>>>>>>>>>>>>>>>>>>>>>>> engine compatibility. It recommends resolving table identifiers at read
>>>>>>>>>>>>>>>>>>>>>>>>>> time (late binding) rather than creation time, and introduces UUID-based
>>>>>>>>>>>>>>>>>>>>>>>>>> validation to maintain identity guarantees across engines, or sessions. It
>>>>>>>>>>>>>>>>>>>>>>>>>> also revises how default-catalog and default-namespace are handled
>>>>>>>>>>>>>>>>>>>>>>>>>> (defaulting both to the session context if not explicitly set) to better
>>>>>>>>>>>>>>>>>>>>>>>>>> align with engine behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have
>>>>>>>>>>>>>>>>>>>>>>>>>> addressed the comments.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog names in the spec refer to logical catalogs—i.e., the first part of
>>>>>>>>>>>>>>>>>>>>>>>>>> a three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung
>>>>>>>>>>>>>>>>>>>>>>>>>> Yun <su...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I
>>>>>>>>>>>>>>>>>>>>>>>>>> think view portability is a very important topic for us to continue
>>>>>>>>>>>>>>>>>>>>>>>>>> discussing as it relies on many assumptions within the data ecosystem for
>>>>>>>>>>>>>>>>>>>>>>>>>> it to function like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how
>>>>>>>>>>>>>>>>>>>>>>>>>> this may impact the permission questions the engines will be asking, and
>>>>>>>>>>>>>>>>>>>>>>>>>> whether that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this
>>>>>>>>>>>>>>>>>>>>>>>>>> problem. I've added a few comments to get a better understanding of how
>>>>>>>>>>>>>>>>>>>>>>>>>> this will look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline
>>>>>>>>>>>>>>>>>>>>>>>>>> with a few community members to better understand how the current spec is
>>>>>>>>>>>>>>>>>>>>>>>>>> being interpreted. Those conversations served as inputs to a new proposal
>>>>>>>>>>>>>>>>>>>>>>>>>> on how table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1].
>>>>>>>>>>>>>>>>>>>>>>>>>> I look forward to your feedback and working together to move this forward
>>>>>>>>>>>>>>>>>>>>>>>>>> so we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Jan Kaul <ja...@mailbox.org.INVALID>]

I have a question regarding the following representation for the storage 
table identifier:

* catalog name: optional
* namespace: optional
* table name: required
* UUID: required

Having optional fields as part of the identifier again raises the 
question of what to use if these optional fields are not set. And we 
again have the early vs late binding discussion. Where early binding 
would default to the view catalog and late binding would use the current 
session catalog/namespace.

@Walaa: Which behavior did you have in mind?

In my opinion, having the functionality of MVs depend on the current 
session context is not desirable. So I would argue against late binding.

Best wishes,

Jan

On 5/8/25 19:59, Walaa Eldin Moustafa wrote:
> Let us continue here for the context and continuity of the conversation.
>
> On Thu, May 8, 2025 at 10:55 AM Steven Wu <st...@gmail.com> wrote:
>
>     For the storage table identifier discussion, can we move to Jan's
>     PR on materialized view spec?
>     https://github.com/apache/iceberg/pull/11041
>
>
>     On Thu, May 8, 2025 at 10:40 AM Walaa Eldin Moustafa
>     <wa...@gmail.com> wrote:
>
>         It looks like we may not reach alignment on the resolution
>         method in the short term. That said, perhaps we can move
>         forward by aligning on the format used to store table
>         identifiers in metadata instead.
>
>         The idea is that the same format could work under either
>         resolution model, which would allow us to make progress
>         independently of that debate.
>
>         Is this storage format of the storage table identifier
>         agreeable to you?
>         * catalog name: optional
>         * namespace: optional
>         * table name: required
>         * UUID: required
>
>         With the condition that the components included in a table
>         identifier in the metadata should match those used in the view
>         text for that table.
>
>         The same structure above could apply to storing child tables
>         or view identifiers as part of lineage, if and when that’s needed.
>
>         For the longer term resolution model, it might make sense to
>         initiate a cross-community discussion.
>
>         Thanks,
>         Walaa.
>
>         On Thu, May 8, 2025 at 9:07 AM Steven Wu
>         <st...@gmail.com> wrote:
>
>             If a table referenced by the view SQL query is deleted and
>             recreated, is the regular view still valid? If yes, it
>             won't be correct to use resolved UUID to validate the name
>             reference. On the materialized view side, we are aligned
>             that UUID is necessary to detect this scenario and mark
>             the refresh state "stale" if a referenced table is
>             re-created.
>
>             > esolution using default-catalog and default-namespace
>             does not provide full determinism,
>
>             With the new proposal, the catalog and namespace depends
>             on the engine's current context. if the user changes the
>             current context (catalog and database) via SET/USE , the
>             view SQL query may resolve to a different table. That is
>             probably a bigger problem.
>
>             With the current view spec, the intention is set in the
>             view creation time. I would rather have the view producer
>             set the explicit intention (than the implicit dependency
>             on the engine context at view consumption phase).
>
>             Also the view SQL can still reference a table with a
>             catalog name component.  Hence the catalog naming still
>             matters.The new proposal won't really solve the naming
>             problem.
>
>             On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa
>             <wa...@gmail.com> wrote:
>
>                 Hi Dan,
>
>                 Thanks for the clarification! I agree that UUIDs
>                 should not be stored in the SQL definition of the view.
>
>                 Just to clarify: the proposal isn't about modifying
>                 the view definition itself, but rather about how table
>                 identifiers are represented in Iceberg view metadata,
>                 which naturally depends on how those identifiers are
>                 resolved.
>
>                 For example, in order to store the "storage table
>                 identifier" in the metadata:
>                 * Should we apply the same resolution rules based on
>                 default-catalog and default-namespace?
>                 * And if that resolution method is considered
>                 deterministic enough, would you be comfortable not
>                 storing the UUID in the storage table identifier metadata?
>
>                 The same questions apply when storing child views or
>                 tables as part of the view’s lineage metadata. It
>                 would be helpful to clarify the resolution model and
>                 whether UUIDs should be part of that for validation or
>                 drift detection.
>
>                 Thanks,
>                 Walaa.
>
>
>
>                 On Thu, May 8, 2025 at 8:29 AM Daniel Weeks
>                 <dw...@apache.org> wrote:
>
>                     I don't think we want to include the resolved
>                     table UUIDs in the view definition, but rather in
>                     the storage table state.  You can still resolve
>                     whether those drift at some point, but I don't
>                     feel like it's a good idea to capture data in
>                     the view that we may allow to drift if there isn't
>                     any requirement that they match.  This also aligns
>                     with the identifier resolution being late binding.
>
>                     -Dan
>
>                     On Wed, May 7, 2025 at 10:45 PM Walaa Eldin
>                     Moustafa <wa...@gmail.com> wrote:
>
>                         Thanks Steven! So would you agree that
>                         resolution using default-catalog and
>                         default-namespace does not provide full
>                         determinism, and requires a supporting safety
>                         mechanism?
>
>                         Thanks,
>                         Walaa.
>
>                         On Wed, May 7, 2025 at 10:30 PM Steven Wu
>                         <st...@gmail.com> wrote:
>
>                             > If the current model is considered
>                             deterministic, do you think
>                             `default-catalog` and `default-namespace`
>                             fields provide enough determinism to
>                             eliminate the need for UUIDs when storing
>                             table identifiers?
>
>                             I am fine with storing UUIDs for table
>                             identifiers in the view. Basically, view
>                             creation resolves all referenced
>                             tables/views with UUIDs. View consumers
>                             can validate resolved tables/views with
>                             the stored UUIDs and fail the query if
>                             mismatch.
>
>                             The UUID change doesn't really change the
>                             table identifier resolution rule though.
>                             It is more of a safety protection.
>
>                             On Wed, May 7, 2025 at 10:02 PM Walaa
>                             Eldin Moustafa <wa...@gmail.com> wrote:
>
>                                 Hi Steven,
>
>                                 Thanks for the reply.
>
>                                 > I agree with Dan that we shouldn't
>                                 solve catalog naming in the Iceberg
>                                 view spec.
>
>                                 To clarify, I don't believe the
>                                 proposal is trying to solve catalog
>                                 naming. What it’s doing is simply this:
>
>                                 * Proposing that table names inside
>                                 views resolve the same way as they do
>                                 elsewhere (e.g., queries).
>                                 * Adopting a model that is already
>                                 widely used and supported in the
>                                 existing ecosystem, which allows for:
>                                     -- Renaming catalog aliases
>                                     -- Swapping catalog
>                                 implementations behind consistent names
>                                     -- Having different default
>                                 catalog names across engines that
>                                 still point to the same underlying tables
>
>                                 These are common patterns in
>                                 production data lakes. Saying Iceberg
>                                 views cannot operate in those
>                                 environments feels unrealistic. In
>                                 practice, it means the spec breaks
>                                 down in situations that users
>                                 encounter regularly.
>
>                                 > The recommendation of using engines’
>                                 current catalog and database can cause
>                                 context-dependent resolution results.
>
>                                 * As noted in the doc and earlier
>                                 replies, fixing a catalog name doesn’t
>                                 actually guarantee determinism either.
>                                 All the failure scenarios above still
>                                 apply even when a default-catalog is
>                                 stored.
>                                 * The current spec also allows
>                                 default-catalog to be null, in which
>                                 case it falls back to the view’s
>                                 catalog, yet that catalog is
>                                 determined based on how the view is
>                                 referenced in the query, which would
>                                 be considered non-deterministic based
>                                 on the same criteria you shared.
>                                 * The only true form of determinism
>                                 here is UUID-based validation, which
>                                 protects against silent drift in any
>                                 resolution model.
>
>                                 If the current model is considered
>                                 deterministic, do you think
>                                 `default-catalog` and
>                                 `default-namespace` fields provide
>                                 enough determinism to eliminate the
>                                 need for UUIDs when storing table
>                                 identifiers?
>                                 Or put another way: Would you be
>                                 comfortable relying solely on
>                                 default-catalog + default-namespace +
>                                 table name to re-identify the correct
>                                 table, without UUID validation?
>
>                                 +1 on involving other communities. I’m
>                                 happy to help facilitate a
>                                 cross-community discussion if we
>                                 aren’t able to reach a resolution here.
>
>                                 Thanks,
>                                 Walaa.
>
>
>
>                                 On Wed, May 7, 2025 at 9:20 PM Steven
>                                 Wu <st...@gmail.com> wrote:
>
>                                     I agree with Dan that we shouldn't
>                                     solve catalog naming in the
>                                     Iceberg view spec. I am not
>                                     convinced that the proposed change
>                                     will make the table identifier
>                                     resolution more clear and
>                                     portable. The recommendation of
>                                     using engines' current catalog and
>                                     database can cause context
>                                     dependent resolution results,
>                                     which seems non-deterministic to me.
>
>                                     Walaa, you raised a point in the
>                                     doc that the current catalog
>                                     resolution logic (default-catalog
>                                     field, then view catalog) is
>                                     challenging and unrealistic for
>                                     engines (like Spark and Trino). It
>                                     will be great to get more inputs
>                                     from the broader community on this
>                                     part.
>
>
>                                     On Tue, May 6, 2025 at 9:21 AM
>                                     Benny Chow <bt...@gmail.com> wrote:
>
>                                         In Spark, I believe that the
>                                         USE commands sets the current
>                                         catalog and namespace.  This
>                                         affects both where the view is
>                                         created and how unqualified
>                                         table identifiers are
>                                         resolved.  I also don't see an
>                                         issue with saving the current
>                                         catalog and namespace into the
>                                         view
>                                         metadata's default-catalog and
>                                         default-namespace fields.
>
>                                         On Wed, Apr 30, 2025 at
>                                         5:12 PM Walaa Eldin Moustafa
>                                         <wa...@gmail.com> wrote:
>
>                                             > I think that's the lesser
>                                             evil compared to Iceberg
>                                             specifying how engines
>                                             should resolve identifiers
>
>                                             I think this is also
>                                             similar to the previous
>                                             point. It is the other way
>                                             around. Right now the spec
>                                             dictates how to resolve
>                                             (through employing a
>                                             view-specific
>                                             `default-catalog` field).
>                                             The proposal is suggesting
>                                             to get out of this space
>                                             and let engines handle it
>                                             similar to how they handle
>                                             all identifiers.
>
>                                             On Wed, Apr 30, 2025 at
>                                             5:07 PM Walaa Eldin
>                                             Moustafa
>                                             <wa...@gmail.com> wrote:
>
>                                                 > I thought
>                                                 "default-catalog"
>                                                 could be set via the
>                                                 USE command.
>
>                                                 Benny, I think this is
>                                                 a misconception or
>                                                 miscommunication. The
>                                                 USE command has no
>                                                 impact on the
>                                                 `default-catalog`
>                                                 field. In fact, the
>                                                 proposal's direction
>                                                 is exactly to
>                                                 establish that USE
>                                                 command should
>                                                 influence how tables
>                                                 are resolved, same
>                                                 like everywhere else.
>                                                 Right now it is not
>                                                 the case under the
>                                                 current spec.
>
>
>                                                 On Wed, Apr 30, 2025
>                                                 at 3:17 PM Benny Chow
>                                                 <bt...@gmail.com> wrote:
>
>                                                     > there is no SQL
>                                                     construct today to
>                                                     explicitly set
>                                                     default-catalog
>
>                                                     I thought
>                                                     "default-catalog"
>                                                     could be set via
>                                                     the USE command.
>
>                                                     I generally agree
>                                                     with Dan about
>                                                     requiring
>                                                     consistent catalog
>                                                     names.  I think
>                                                     that's the lesser
>                                                     evil compared to
>                                                     Iceberg specifying
>                                                     how engines should
>                                                     resolve
>                                                     identifiers.
>                                                     Another thing to
>                                                     consider is that
>                                                     identifier
>                                                     resolution can be
>                                                     very expensive at
>                                                     query validation
>                                                     time if
>                                                     identifiers need
>                                                     to be looked up
>                                                     from a bunch of
>                                                     places. Hopefully,
>                                                     it should be
>                                                     possible to define
>                                                     a view in such a
>                                                     way that
>                                                     identifiers can be
>                                                     resolved on the
>                                                     first try.
>
>                                                     Benny
>
>                                                     On Tue, Apr 29,
>                                                     2025 at 10:29 PM
>                                                     Walaa Eldin
>                                                     Moustafa
>                                                     <wa...@gmail.com>
>                                                     wrote:
>
>                                                         Hi Rishabh,
>
>                                                         You're right
>                                                         that the
>                                                         proposal
>                                                         touches on two
>                                                         aspects, and
>                                                         resolution
>                                                         rules are one
>                                                         of them. The
>                                                         other aspect
>                                                         is the
>                                                         proposal's
>                                                         position that
>                                                         table
>                                                         identifiers
>                                                         should be
>                                                         stored in
>                                                         metadata
>                                                         exactly as
>                                                         they appear in
>                                                         the view text
>                                                         (e.g., even if
>                                                         they're
>                                                         two-part or
>                                                         partially
>                                                         qualified),
>                                                         along with
>                                                         their
>                                                         corresponding
>                                                         UUIDs for
>                                                         validation.
>                                                         This applies
>                                                         both to
>                                                         referenced
>                                                         input tables
>                                                         and the
>                                                         storage table
>                                                         identifier in
>                                                         materialized
>                                                         views.
>
>                                                         We may be able
>                                                         to converge on
>                                                         this storage
>                                                         format even if
>                                                         we haven't yet
>                                                         converged on
>                                                         the resolution
>                                                         fallback
>                                                         rules. I
>                                                         believe both
>                                                         resolution
>                                                         strategies
>                                                         currently
>                                                         being
>                                                         discussed
>                                                         would still
>                                                         lead to
>                                                         storing
>                                                         identifiers in
>                                                         this way.
>
>                                                         I'm supportive
>                                                         of moving
>                                                         forward with
>                                                         consensus on
>                                                         the identifier
>                                                         storage
>                                                         format. That
>                                                         said, we may
>                                                         continue to
>                                                         run into
>                                                         questions
>                                                         related to
>                                                         resolution
>                                                         during
>                                                         implementation.
>                                                         For example:
>                                                         Should the
>                                                         storage table
>                                                         identifier
>                                                         follow the
>                                                         same
>                                                         default-catalog
>                                                         and
>                                                         default-namespace
>                                                         resolution
>                                                         behavior as
>                                                         other table
>                                                         references?
>
>                                                         Thanks,
>                                                         Walaa.
>
>                                                         On Tue, Apr
>                                                         29, 2025 at
>                                                         10:07 PM
>                                                         Rishabh Bhatia
>                                                         <bh...@gmail.com>
>                                                         wrote:
>
>                                                             Hello Walaa,
>
>                                                             Thanks for
>                                                             starting
>                                                             this
>                                                             discussion.
>
>                                                             I think we
>                                                             should
>                                                             decouple
>                                                             at least
>                                                             the MV
>                                                             Spec from
>                                                             the
>                                                             proposal
>                                                             to change
>                                                             the
>                                                             current
>                                                             behavior
>                                                             of view
>                                                             resolution.
>
>                                                             We can
>                                                             continue
>                                                             having the
>                                                             discussion
>                                                             if the
>                                                             current
>                                                             view spec
>                                                             needs to
>                                                             be changed
>                                                             or not.
>                                                             Based on
>                                                             the
>                                                             decision
>                                                             at a later
>                                                             point if
>                                                             required
>                                                             we can
>                                                             update the
>                                                             view
>                                                             resolution
>                                                             rule.
>
>
>                                                             Thanks,
>                                                             Rishabh
>
>                                                             On Mon,
>                                                             Apr 28,
>                                                             2025 at
>                                                             3:22 PM
>                                                             Walaa
>                                                             Eldin
>                                                             Moustafa
>                                                             <wa...@gmail.com>
>                                                             wrote:
>
>                                                                 Correction
>                                                                 of
>                                                                 typo:
>                                                                 both
>                                                                 engines
>                                                                 seem
>                                                                 to set
>                                                                 default-catalog
>                                                                 to the
>                                                                 view
>                                                                 catalog
>                                                                 if it
>                                                                 is
>                                                                 defined,
>                                                                 or to
>                                                                 null
>                                                                 if the
>                                                                 view
>                                                                 catalog
>                                                                 is not
>                                                                 defined.
>
>                                                                 On
>                                                                 Mon,
>                                                                 Apr
>                                                                 28,
>                                                                 2025
>                                                                 at
>                                                                 3:06 PM
>                                                                 Walaa
>                                                                 Eldin
>                                                                 Moustafa
>                                                                 <wa...@gmail.com>
>                                                                 wrote:
>
>                                                                     Hi
>                                                                     Dan,
>
>                                                                     Thanks
>                                                                     again
>                                                                     for
>                                                                     your
>                                                                     response.
>
>                                                                     I
>                                                                     agree
>                                                                     that
>                                                                     catalog
>                                                                     renaming
>                                                                     is
>                                                                     an
>                                                                     environmental
>                                                                     event,
>                                                                     but
>                                                                     it's
>                                                                     a
>                                                                     real
>                                                                     one
>                                                                     that
>                                                                     happens
>                                                                     frequently
>                                                                     in
>                                                                     practice.
>                                                                     Saying
>                                                                     that
>                                                                     the
>                                                                     Iceberg
>                                                                     spec
>                                                                     cannot
>                                                                     accommodate
>                                                                     something
>                                                                     as
>                                                                     common
>                                                                     as
>                                                                     catalog
>                                                                     renaming
>                                                                     feels
>                                                                     very
>                                                                     restrictive,
>                                                                     and
>                                                                     could
>                                                                     make
>                                                                     the
>                                                                     spec
>                                                                     less
>                                                                     practical,
>                                                                     even
>                                                                     unusable,
>                                                                     for
>                                                                     real-world
>                                                                     deployments.
>                                                                     I’m
>                                                                     sharing
>                                                                     this
>                                                                     from
>                                                                     the
>                                                                     perspective
>                                                                     of
>                                                                     a
>                                                                     large
>                                                                     data
>                                                                     lake
>                                                                     environment
>                                                                     where
>                                                                     views
>                                                                     are
>                                                                     heavily
>                                                                     deployed
>                                                                     and
>                                                                     operationalized.
>
>                                                                     Further,
>                                                                     it's
>                                                                     worth
>                                                                     noting
>                                                                     that
>                                                                     the
>                                                                     table
>                                                                     spec
>                                                                     is
>                                                                     resilient
>                                                                     to
>                                                                     catalog
>                                                                     renaming,
>                                                                     but
>                                                                     the
>                                                                     view
>                                                                     spec
>                                                                     is
>                                                                     not.
>                                                                     If
>                                                                     we
>                                                                     have
>                                                                     an
>                                                                     opportunity
>                                                                     to
>                                                                     make
>                                                                     the
>                                                                     view
>                                                                     spec
>                                                                     similarly
>                                                                     resilient,
>                                                                     I
>                                                                     wonder
>                                                                     why
>                                                                     not?
>                                                                     Both
>                                                                     specifications
>                                                                     are
>                                                                     deterministic
>                                                                     in
>                                                                     their
>                                                                     definition,
>                                                                     but
>                                                                     one
>                                                                     is
>                                                                     more
>                                                                     fragile
>                                                                     to
>                                                                     environmental
>                                                                     changes
>                                                                     than
>                                                                     the
>                                                                     other.
>                                                                     Improving
>                                                                     resilience
>                                                                     does
>                                                                     not
>                                                                     sacrifice
>                                                                     determinism.
>                                                                     It
>                                                                     simply
>                                                                     makes
>                                                                     views
>                                                                     safer
>                                                                     and
>                                                                     more
>                                                                     portable
>                                                                     over
>                                                                     time.
>
>                                                                     Separately,
>                                                                     given
>                                                                     that
>                                                                     there
>                                                                     is
>                                                                     no
>                                                                     SQL
>                                                                     construct
>                                                                     today
>                                                                     to
>                                                                     explicitly
>                                                                     set
>                                                                     default-catalog
>                                                                     at
>                                                                     creation
>                                                                     time, what
>                                                                     is
>                                                                     the
>                                                                     intuition
>                                                                     behind
>                                                                     how
>                                                                     engines
>                                                                     like
>                                                                     Spark
>                                                                     and
>                                                                     Trino
>                                                                     currently
>                                                                     assign
>                                                                     default-catalog?
>                                                                     Today,
>                                                                     both
>                                                                     engines
>                                                                     seem
>                                                                     to
>                                                                     set
>                                                                     default-catalog
>                                                                     to
>                                                                     null
>                                                                     if
>                                                                     the
>                                                                     view
>                                                                     catalog
>                                                                     is
>                                                                     defined,
>                                                                     or
>                                                                     to
>                                                                     the
>                                                                     view
>                                                                     catalog
>                                                                     if
>                                                                     not.
>                                                                     What
>                                                                     was
>                                                                     the
>                                                                     intended
>                                                                     thought
>                                                                     process
>                                                                     behind
>                                                                     this
>                                                                     behavior?
>
>                                                                     Thanks,
>                                                                     Walaa
>
>                                                                     On
>                                                                     Mon,
>                                                                     Apr
>                                                                     28,
>                                                                     2025
>                                                                     at
>                                                                     1:33 PM
>                                                                     Daniel
>                                                                     Weeks
>                                                                     <dw...@apache.org>
>                                                                     wrote:
>
>                                                                         Walaa,
>
>
>                                                                         >
>                                                                         tables
>                                                                         inside
>                                                                         views
>                                                                         remain
>                                                                         reachable
>                                                                         after
>                                                                         a
>                                                                         catalog
>                                                                         rename
>
>                                                                         This
>                                                                         problem
>                                                                         stems
>                                                                         from
>                                                                         the exact
>                                                                         environmental/configuration
>                                                                         issue
>                                                                         that
>                                                                         we
>                                                                         should
>                                                                         not
>                                                                         be
>                                                                         trying
>                                                                         to
>                                                                         address. 
>                                                                         I
>                                                                         don't
>                                                                         think
>                                                                         we
>                                                                         would
>                                                                         expect
>                                                                         references
>                                                                         to
>                                                                         survive
>                                                                         a
>                                                                         catalog
>                                                                         rename.
>                                                                         That's
>                                                                         not
>                                                                         something
>                                                                         covered
>                                                                         by
>                                                                         the
>                                                                         spec
>                                                                         and
>                                                                         needs
>                                                                         to
>                                                                         be
>                                                                         handled
>                                                                         separately
>                                                                         as
>                                                                         a
>                                                                         platform-level
>                                                                         migration
>                                                                         specific
>                                                                         to
>                                                                         the
>                                                                         affected
>                                                                         environment.
>
>                                                                         The
>                                                                         identifier
>                                                                         resolution
>                                                                         logic
>                                                                         is
>                                                                         clear
>                                                                         and
>                                                                         deterministic.
>                                                                         It
>                                                                         should
>                                                                         not
>                                                                         matter
>                                                                         whether
>                                                                         an
>                                                                         engine
>                                                                         resolves
>                                                                         and
>                                                                         encodes
>                                                                         the
>                                                                         default-catalog
>                                                                         or
>                                                                         leaves
>                                                                         it
>                                                                         to
>                                                                         the
>                                                                         resolution
>                                                                         rules.
>
>                                                                         The
>                                                                         issue
>                                                                         isn't
>                                                                         with
>                                                                         how
>                                                                         the
>                                                                         spec
>                                                                         is
>                                                                         defined,
>                                                                         but
>                                                                         rather
>                                                                         view
>                                                                         behavior
>                                                                         when
>                                                                         you
>                                                                         start
>                                                                         altering
>                                                                         the
>                                                                         environment
>                                                                         around
>                                                                         it,
>                                                                         which
>                                                                         isn't
>                                                                         something we
>                                                                         should
>                                                                         be
>                                                                         trying
>                                                                         to
>                                                                         define
>                                                                         here.
>
>                                                                         -Dan
>
>                                                                         On
>                                                                         Mon,
>                                                                         Apr
>                                                                         28,
>                                                                         2025
>                                                                         at
>                                                                         12:17 PM
>                                                                         Walaa
>                                                                         Eldin
>                                                                         Moustafa
>                                                                         <wa...@gmail.com>
>                                                                         wrote:
>
>                                                                             Hi
>                                                                             Dan,
>
>                                                                             Thanks
>                                                                             for
>                                                                             chiming
>                                                                             in.
>
>                                                                             I
>                                                                             believe
>                                                                             the
>                                                                             issues
>                                                                             we’re
>                                                                             seeing
>                                                                             now
>                                                                             go
>                                                                             beyond
>                                                                             just
>                                                                             catalog
>                                                                             naming
>                                                                             consistency.
>                                                                             The
>                                                                             behavior
>                                                                             around
>                                                                             default-catalog
>                                                                             itself
>                                                                             introduces
>                                                                             resolution
>                                                                             inconsistencies
>                                                                             even
>                                                                             when
>                                                                             catalog
>                                                                             names
>                                                                             are
>                                                                             consistent.
>                                                                             For
>                                                                             example:
>
>                                                                             *
>                                                                             When
>                                                                             default-catalog
>                                                                             is
>                                                                             set
>                                                                             to
>                                                                             null,
>                                                                             tables
>                                                                             inside
>                                                                             views
>                                                                             remain
>                                                                             reachable
>                                                                             after
>                                                                             a
>                                                                             catalog
>                                                                             rename.
>                                                                             But
>                                                                             if
>                                                                             it
>                                                                             is
>                                                                             set
>                                                                             to
>                                                                             a
>                                                                             non-null
>                                                                             value,
>                                                                             table
>                                                                             references
>                                                                             will
>                                                                             break.
>
>                                                                             *
>                                                                             default-catalog
>                                                                             causes
>                                                                             table
>                                                                             references
>                                                                             inside
>                                                                             views
>                                                                             to
>                                                                             be
>                                                                             early
>                                                                             bound
>                                                                             (i.e.,
>                                                                             bound
>                                                                             at
>                                                                             view
>                                                                             creation
>                                                                             time,
>                                                                             especially
>                                                                             when
>                                                                             using
>                                                                             a
>                                                                             non-null
>                                                                             value),
>                                                                             while
>                                                                             table
>                                                                             references
>                                                                             inside
>                                                                             standalone
>                                                                             queries
>                                                                             are
>                                                                             late
>                                                                             bound
>                                                                             (bound
>                                                                             at
>                                                                             query
>                                                                             time).
>                                                                             This
>                                                                             creates
>                                                                             inconsistencies
>                                                                             when
>                                                                             resolving
>                                                                             the
>                                                                             same
>                                                                             table
>                                                                             name
>                                                                             inside
>                                                                             and
>                                                                             outside
>                                                                             views,
>                                                                             even
>                                                                             within
>                                                                             the
>                                                                             same
>                                                                             job.
>
>                                                                             *
>                                                                             It
>                                                                             causes
>                                                                             Spark's
>                                                                             and
>                                                                             Trino
>                                                                             behavior
>                                                                             to
>                                                                             drift
>                                                                             from
>                                                                             the
>                                                                             spec.
>                                                                             There
>                                                                             is
>                                                                             no
>                                                                             way
>                                                                             to
>                                                                             fully
>                                                                             align
>                                                                             Spark's
>                                                                             behavior
>                                                                             without
>                                                                             making
>                                                                             invasive
>                                                                             changes
>                                                                             to
>                                                                             the
>                                                                             Spark
>                                                                             SQL
>                                                                             grammar
>                                                                             and
>                                                                             the
>                                                                             View
>                                                                             DataSource
>                                                                             API
>                                                                             (specifically
>                                                                             on
>                                                                             the
>                                                                             CREATE
>                                                                             side).
>                                                                             This
>                                                                             challenge
>                                                                             would
>                                                                             extend
>                                                                             to
>                                                                             other
>                                                                             engines
>                                                                             too.
>                                                                             Both
>                                                                             Spark
>                                                                             and
>                                                                             Trino
>                                                                             set
>                                                                             this
>                                                                             field
>                                                                             based
>                                                                             on
>                                                                             a
>                                                                             heuristic
>                                                                             in
>                                                                             today's
>                                                                             implementation.
>
>                                                                             *
>                                                                             With
>                                                                             view
>                                                                             nesting
>                                                                             (views
>                                                                             depending
>                                                                             on
>                                                                             views),
>                                                                             these
>                                                                             inconsistencies
>                                                                             amplify
>                                                                             further,
>                                                                             forcing
>                                                                             users
>                                                                             and
>                                                                             engines
>                                                                             to
>                                                                             reason
>                                                                             about
>                                                                             catalog
>                                                                             resolution
>                                                                             at
>                                                                             every
>                                                                             level
>                                                                             in
>                                                                             the
>                                                                             view
>                                                                             tree.
>
>                                                                             *
>                                                                             It
>                                                                             will
>                                                                             be
>                                                                             difficult
>                                                                             to
>                                                                             migrate
>                                                                             Hive
>                                                                             views
>                                                                             to
>                                                                             Iceberg
>                                                                             with
>                                                                             that
>                                                                             model.
>                                                                             Migrated
>                                                                             Hive
>                                                                             views
>                                                                             will
>                                                                             have
>                                                                             to
>                                                                             unfollow
>                                                                             that
>                                                                             spec.
>
>                                                                             How
>                                                                             would
>                                                                             you
>                                                                             suggest
>                                                                             approaching
>                                                                             the
>                                                                             engine-level
>                                                                             changes
>                                                                             required
>                                                                             to
>                                                                             support
>                                                                             the
>                                                                             current
>                                                                             default-catalog
>                                                                             field?
>                                                                             Also,
>                                                                             do
>                                                                             you
>                                                                             believe
>                                                                             the
>                                                                             Spark
>                                                                             and
>                                                                             Trino
>                                                                             communities
>                                                                             would
>                                                                             align
>                                                                             around
>                                                                             having
>                                                                             table
>                                                                             resolution
>                                                                             behave
>                                                                             inconsistently
>                                                                             between
>                                                                             queries
>                                                                             and
>                                                                             views, or
>                                                                             inconsistency
>                                                                             between
>                                                                             Iceberg
>                                                                             and
>                                                                             other
>                                                                             types
>                                                                             of
>                                                                             views?
>
>                                                                             Thanks,
>                                                                             Walaa
>
>
>
>                                                                             On
>                                                                             Mon,
>                                                                             Apr
>                                                                             28,
>                                                                             2025
>                                                                             at
>                                                                             11:34 AM
>                                                                             Daniel
>                                                                             Weeks
>                                                                             <dw...@apache.org>
>                                                                             wrote:
>
>                                                                                 I
>                                                                                 would
>                                                                                 agree
>                                                                                 with
>                                                                                 Jan's
>                                                                                 summary
>                                                                                 of
>                                                                                 why
>                                                                                 'default-catalog'
>                                                                                 was
>                                                                                 introduced,
>                                                                                 but
>                                                                                 I
>                                                                                 think
>                                                                                 we
>                                                                                 need
>                                                                                 to
>                                                                                 step
>                                                                                 back
>                                                                                 and
>                                                                                 align
>                                                                                 on
>                                                                                 what
>                                                                                 we
>                                                                                 are
>                                                                                 really
>                                                                                 attempting
>                                                                                 to
>                                                                                 support
>                                                                                 in
>                                                                                 the
>                                                                                 spec.
>
>
>                                                                                 The
>                                                                                 issues
>                                                                                 we're
>                                                                                 discussing
>                                                                                 largely
>                                                                                 stem
>                                                                                 from
>                                                                                 using
>                                                                                 multiple
>                                                                                 engines
>                                                                                 with
>                                                                                 cross
>                                                                                 catalog
>                                                                                 references
>                                                                                 and
>                                                                                 configurations
>                                                                                 where
>                                                                                 catalog
>                                                                                 names
>                                                                                 are
>                                                                                 not
>                                                                                 aligned. 
>                                                                                 If
>                                                                                 we
>                                                                                 have
>                                                                                 multiple
>                                                                                 engines
>                                                                                 that
>                                                                                 all
>                                                                                 have
>                                                                                 the
>                                                                                 same
>                                                                                 catalog
>                                                                                 names/configurations,
>                                                                                 the
>                                                                                 current
>                                                                                 spec
>                                                                                 implementation
>                                                                                 is
>                                                                                 well
>                                                                                 defined
>                                                                                 for
>                                                                                 table
>                                                                                 resolution
>                                                                                 even
>                                                                                 across
>                                                                                 catalogs. 
>                                                                                 The
>                                                                                 'default-catalog'
>                                                                                 (and
>                                                                                 namespace
>                                                                                 equivalent)
>                                                                                 was
>                                                                                 intended
>                                                                                 to
>                                                                                 address
>                                                                                 the
>                                                                                 resolution
>                                                                                 within
>                                                                                 the
>                                                                                 context
>                                                                                 of
>                                                                                 the
>                                                                                 sql
>                                                                                 text,
>                                                                                 not
>                                                                                 to
>                                                                                 address
>                                                                                 catalog/naming
>                                                                                 inconsistencies.
>
>                                                                                 I
>                                                                                 feel
>                                                                                 like
>                                                                                 we're
>                                                                                 trying
>                                                                                 to
>                                                                                 adapt the
>                                                                                 original
>                                                                                 intent
>                                                                                 to
>                                                                                 address
>                                                                                 the
>                                                                                 catalog
>                                                                                 naming/configuration
>                                                                                 and
>                                                                                 would
>                                                                                 argue
>                                                                                 that
>                                                                                 we
>                                                                                 shouldn't
>                                                                                 attempt
>                                                                                 to
>                                                                                 do
>                                                                                 that
>                                                                                 as
>                                                                                 part
>                                                                                 of
>                                                                                 the
>                                                                                 spec.
>                                                                                 Inconsistently
>                                                                                 named
>                                                                                 catalogs
>                                                                                 are
>                                                                                 a
>                                                                                 reality,
>                                                                                 but
>                                                                                 we
>                                                                                 should consider
>                                                                                 that
>                                                                                 a
>                                                                                 configuration/environmental
>                                                                                 issue,
>                                                                                 not
>                                                                                 something
>                                                                                 to
>                                                                                 solve
>                                                                                 for
>                                                                                 in
>                                                                                 the
>                                                                                 spec.
>
>
>                                                                                 We
>                                                                                 should
>                                                                                 support
>                                                                                 and
>                                                                                 advocate
>                                                                                 for
>                                                                                 consistency
>                                                                                 in
>                                                                                 catalog
>                                                                                 naming
>                                                                                 and
>                                                                                 define
>                                                                                 the
>                                                                                 spec
>                                                                                 along
>                                                                                 those
>                                                                                 lines.
>                                                                                 The
>                                                                                 fact
>                                                                                 is
>                                                                                 that
>                                                                                 with
>                                                                                 all
>                                                                                 of
>                                                                                 the
>                                                                                 recent
>                                                                                 work
>                                                                                 that's
>                                                                                 gone
>                                                                                 into
>                                                                                 making
>                                                                                 catalogs
>                                                                                 pluggable,
>                                                                                 it
>                                                                                 makes
>                                                                                 more
>                                                                                 sense
>                                                                                 to
>                                                                                 just
>                                                                                 register
>                                                                                 catalog
>                                                                                 configuration
>                                                                                 with
>                                                                                 consistent
>                                                                                 names
>                                                                                 (even
>                                                                                 if
>                                                                                 you
>                                                                                 have
>                                                                                 to
>                                                                                 duplicate
>                                                                                 the
>                                                                                 configuration
>                                                                                 for
>                                                                                 supporting
>                                                                                 existing
>                                                                                 readers/writers).
>                                                                                 I
>                                                                                 think
>                                                                                 it's
>                                                                                 better
>                                                                                 to
>                                                                                 provide
>                                                                                 a
>                                                                                 path
>                                                                                 toward
>                                                                                 consistency
>                                                                                 than
>                                                                                 to
>                                                                                 normalize
>                                                                                 complicated
>                                                                                 schemes
>                                                                                 to
>                                                                                 workaround
>                                                                                 the
>                                                                                 issues
>                                                                                 caused
>                                                                                 by
>                                                                                 environmental/configuration
>                                                                                 inconsistencies.
>
>                                                                                 If
>                                                                                 the
>                                                                                 goal
>                                                                                 is
>                                                                                 to
>                                                                                 create
>                                                                                 clever
>                                                                                 ways
>                                                                                 to
>                                                                                 hack
>                                                                                 the
>                                                                                 late
>                                                                                 binding
>                                                                                 resolution
>                                                                                 to
>                                                                                 swap
>                                                                                 in
>                                                                                 different
>                                                                                 catalogs
>                                                                                 or
>                                                                                 make
>                                                                                 references
>                                                                                 contextual,
>                                                                                 I
>                                                                                 feel
>                                                                                 like
>                                                                                 that
>                                                                                 is
>                                                                                 something
>                                                                                 we
>                                                                                 should
>                                                                                 strongly
>                                                                                 discourage
>                                                                                 as
>                                                                                 it
>                                                                                 leads
>                                                                                 to
>                                                                                 confusion
>                                                                                 about
>                                                                                 what
>                                                                                 is
>                                                                                 resolved
>                                                                                 as
>                                                                                 part
>                                                                                 of
>                                                                                 the
>                                                                                 query.
>
>                                                                                 At
>                                                                                 this
>                                                                                 point,
>                                                                                 I
>                                                                                 don't
>                                                                                 see
>                                                                                 a
>                                                                                 good
>                                                                                 argument
>                                                                                 to
>                                                                                 add
>                                                                                 additional configuration
>                                                                                 or
>                                                                                 change
>                                                                                 the
>                                                                                 resolution
>                                                                                 behaviors.
>
>                                                                                 -Dan
>
>
>
>
>                                                                                 On
>                                                                                 Mon,
>                                                                                 Apr
>                                                                                 28,
>                                                                                 2025
>                                                                                 at
>                                                                                 12:40 AM
>                                                                                 Jan
>                                                                                 Kaul
>                                                                                 <ja...@mailbox.org.invalid>
>                                                                                 wrote:
>
>                                                                                     I
>                                                                                     think
>                                                                                     the
>                                                                                     intention
>                                                                                     with
>                                                                                     the
>                                                                                     "default-catalog"
>                                                                                     was
>                                                                                     that
>                                                                                     every
>                                                                                     query
>                                                                                     engine
>                                                                                     uses
>                                                                                     it
>                                                                                     to
>                                                                                     store
>                                                                                     its
>                                                                                     session
>                                                                                     default
>                                                                                     catalog
>                                                                                     at
>                                                                                     the
>                                                                                     time
>                                                                                     of
>                                                                                     creating
>                                                                                     the
>                                                                                     view.
>                                                                                     This
>                                                                                     way
>                                                                                     the
>                                                                                     view
>                                                                                     could
>                                                                                     be
>                                                                                     reused
>                                                                                     in
>                                                                                     another
>                                                                                     session.
>                                                                                     The
>                                                                                     idea
>                                                                                     was
>                                                                                     not
>                                                                                     to
>                                                                                     introduce
>                                                                                     an
>                                                                                     additional
>                                                                                     SQL
>                                                                                     syntax
>                                                                                     to
>                                                                                     set
>                                                                                     the
>                                                                                     default-catalog.
>
>                                                                                     Generally
>                                                                                     we
>                                                                                     have
>                                                                                     different
>                                                                                     environments
>                                                                                     we
>                                                                                     want
>                                                                                     to
>                                                                                     support
>                                                                                     with
>                                                                                     the
>                                                                                     view
>                                                                                     spec:
>
>                                                                                     1.
>                                                                                     Consistent
>                                                                                     catalog
>                                                                                     naming
>
>                                                                                     When
>                                                                                     the
>                                                                                     environment
>                                                                                     supports
>                                                                                     it,
>                                                                                     using
>                                                                                     consistent
>                                                                                     catalog
>                                                                                     names
>                                                                                     can
>                                                                                     have
>                                                                                     a
>                                                                                     great
>                                                                                     benefit
>                                                                                     for
>                                                                                     multi-catalog,
>                                                                                     multi-engine
>                                                                                     setups.
>                                                                                     With
>                                                                                     consistent
>                                                                                     catalog
>                                                                                     names,
>                                                                                     using
>                                                                                     the
>                                                                                     "default-catalog"
>                                                                                     field
>                                                                                     works
>                                                                                     without
>                                                                                     any
>                                                                                     issues.
>
>                                                                                     2.
>                                                                                     Inconsistent
>                                                                                     catalog
>                                                                                     naming
>
>                                                                                     This
>                                                                                     can
>                                                                                     be
>                                                                                     the
>                                                                                     case
>                                                                                     when
>                                                                                     different
>                                                                                     query
>                                                                                     engines
>                                                                                     refer
>                                                                                     to
>                                                                                     the
>                                                                                     same
>                                                                                     physical
>                                                                                     catalog
>                                                                                     by
>                                                                                     different
>                                                                                     names.
>                                                                                     This
>                                                                                     often
>                                                                                     happens
>                                                                                     because
>                                                                                     different
>                                                                                     query
>                                                                                     engines
>                                                                                     use
>                                                                                     different
>                                                                                     strategies
>                                                                                     to
>                                                                                     setup
>                                                                                     the
>                                                                                     catalogs.
>                                                                                     If
>                                                                                     catalogs
>                                                                                     have
>                                                                                     inconsistent
>                                                                                     naming,
>                                                                                     using
>                                                                                     the
>                                                                                     "default-catalog"
>                                                                                     field
>                                                                                     does
>                                                                                     not
>                                                                                     work
>                                                                                     because
>                                                                                     it
>                                                                                     is
>                                                                                     not
>                                                                                     guaranteed
>                                                                                     that
>                                                                                     the
>                                                                                     catalog
>                                                                                     name
>                                                                                     can
>                                                                                     be
>                                                                                     resolved
>                                                                                     with
>                                                                                     another
>                                                                                     engine.
>                                                                                     Using
>                                                                                     the
>                                                                                     "view
>                                                                                     catalog"
>                                                                                     as
>                                                                                     a
>                                                                                     fallback
>                                                                                     is
>                                                                                     a
>                                                                                     better
>                                                                                     solution
>                                                                                     for
>                                                                                     this
>                                                                                     use
>                                                                                     case,
>                                                                                     as
>                                                                                     it
>                                                                                     avoids
>                                                                                     catalog
>                                                                                     names
>                                                                                     altogether.
>                                                                                     It
>                                                                                     is
>                                                                                     however
>                                                                                     limited
>                                                                                     to
>                                                                                     table
>                                                                                     references
>                                                                                     in
>                                                                                     the
>                                                                                     same
>                                                                                     catalog.
>
>
>                                                                                     What
>                                                                                     do
>                                                                                     you
>                                                                                     think
>                                                                                     of
>                                                                                     introducing
>                                                                                     a
>                                                                                     view
>                                                                                     property
>                                                                                     that
>                                                                                     specifies
>                                                                                     if
>                                                                                     the
>                                                                                     "default-catalog"
>                                                                                     or
>                                                                                     the
>                                                                                     "view
>                                                                                     catalog"
>                                                                                     should
>                                                                                     be
>                                                                                     used?
>                                                                                     This
>                                                                                     way,
>                                                                                     you
>                                                                                     could
>                                                                                     use
>                                                                                     the
>                                                                                     "default-catalog"
>                                                                                     in
>                                                                                     environments
>                                                                                     where
>                                                                                     you
>                                                                                     can
>                                                                                     guarantee
>                                                                                     consistent
>                                                                                     naming,
>                                                                                     but
>                                                                                     you
>                                                                                     would
>                                                                                     be
>                                                                                     able
>                                                                                     to
>                                                                                     directly
>                                                                                     fallback
>                                                                                     to
>                                                                                     the
>                                                                                     "view-catalog"
>                                                                                     when
>                                                                                     you
>                                                                                     don't
>                                                                                     have
>                                                                                     consistent
>                                                                                     naming.
>                                                                                     The
>                                                                                     query
>                                                                                     engines
>                                                                                     could
>                                                                                     set
>                                                                                     the
>                                                                                     default
>                                                                                     for
>                                                                                     this
>                                                                                     view
>                                                                                     property
>                                                                                     at
>                                                                                     creation
>                                                                                     time.
>                                                                                     Spark
>                                                                                     for
>                                                                                     example
>                                                                                     could
>                                                                                     set
>                                                                                     it
>                                                                                     to
>                                                                                     automatically
>                                                                                     use
>                                                                                     the
>                                                                                     "view
>                                                                                     catalog".
>
>                                                                                     Thanks
>
>                                                                                     Jan
>
>
>                                                                                     On
>                                                                                     4/26/25
>                                                                                     05:33,
>                                                                                     Walaa
>                                                                                     Eldin
>                                                                                     Moustafa
>                                                                                     wrote:
>>                                                                                     To
>>                                                                                     help
>>                                                                                     folks
>>                                                                                     catch
>>                                                                                     up
>>                                                                                     on
>>                                                                                     the
>>                                                                                     latest
>>                                                                                     discussions
>>                                                                                     and
>>                                                                                     interpretation
>>                                                                                     of
>>                                                                                     the
>>                                                                                     spec,
>>                                                                                     I
>>                                                                                     have
>>                                                                                     summarized
>>                                                                                     everything
>>                                                                                     we
>>                                                                                     discussed
>>                                                                                     so
>>                                                                                     far
>>                                                                                     at
>>                                                                                     the
>>                                                                                     top
>>                                                                                     of
>>                                                                                     the
>>                                                                                     proposal
>>                                                                                     document
>>                                                                                     (here
>>                                                                                     <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>                                                                                     I
>>                                                                                     have
>>                                                                                     slightly
>>                                                                                     updated
>>                                                                                     the
>>                                                                                     proposal
>>                                                                                     to
>>                                                                                     be
>>                                                                                     in
>>                                                                                     sync
>>                                                                                     with
>>                                                                                     the
>>                                                                                     new
>>                                                                                     interpretation
>>                                                                                     to
>>                                                                                     avoid
>>                                                                                     confusion.
>>                                                                                     In
>>                                                                                     summary:
>>
>>                                                                                     *
>>                                                                                     Remove
>>                                                                                     default-catalog
>>                                                                                     and
>>                                                                                     default-namespace
>>                                                                                     fields
>>                                                                                     from
>>                                                                                     the
>>                                                                                     view
>>                                                                                     spec
>>                                                                                     completely.
>>
>>                                                                                     *
>>                                                                                     Hence,
>>                                                                                     we
>>                                                                                     do
>>                                                                                     not
>>                                                                                     attempt
>>                                                                                     to
>>                                                                                     define
>>                                                                                     separate
>>                                                                                     view-level
>>                                                                                     default
>>                                                                                     catalogs
>>                                                                                     or
>>                                                                                     namespaces.
>>
>>                                                                                     Instead:
>>
>>                                                                                     *
>>                                                                                     If
>>                                                                                     a
>>                                                                                     table
>>                                                                                     identifier
>>                                                                                     inside
>>                                                                                     a
>>                                                                                     view
>>                                                                                     lacks
>>                                                                                     a
>>                                                                                     catalog
>>                                                                                     qualifier,
>>                                                                                     engines
>>                                                                                     should
>>                                                                                     resolve
>>                                                                                     it
>>                                                                                     using
>>                                                                                     the
>>                                                                                     current
>>                                                                                     engine
>>                                                                                     catalog
>>                                                                                     at
>>                                                                                     query
>>                                                                                     time.
>>
>>                                                                                     *
>>                                                                                     Reference
>>                                                                                     table
>>                                                                                     identifiers
>>                                                                                     in
>>                                                                                     the
>>                                                                                     metadata
>>                                                                                     exactly
>>                                                                                     as
>>                                                                                     they
>>                                                                                     appear
>>                                                                                     in
>>                                                                                     the
>>                                                                                     view
>>                                                                                     SQL
>>                                                                                     text.
>>
>>                                                                                     *
>>                                                                                     If
>>                                                                                     an
>>                                                                                     identifier
>>                                                                                     lacks
>>                                                                                     the
>>                                                                                     catalog
>>                                                                                     part
>>                                                                                     at
>>                                                                                     creation,
>>                                                                                     it
>>                                                                                     should
>>                                                                                     still
>>                                                                                     lack
>>                                                                                     a
>>                                                                                     catalog
>>                                                                                     in
>>                                                                                     the
>>                                                                                     stored
>>                                                                                     metadata.
>>
>>                                                                                     *
>>                                                                                     Store
>>                                                                                     UUIDs
>>                                                                                     alongside
>>                                                                                     table
>>                                                                                     identifiers
>>                                                                                     whenever
>>                                                                                     possible.
>>
>>                                                                                     Thanks,
>>                                                                                     Walaa.
>>
>>
>>                                                                                     On
>>                                                                                     Fri,
>>                                                                                     Apr
>>                                                                                     25,
>>                                                                                     2025
>>                                                                                     at
>>                                                                                     5:18 PM
>>                                                                                     Walaa
>>                                                                                     Eldin
>>                                                                                     Moustafa
>>                                                                                     <wa...@gmail.com>
>>                                                                                     wrote:
>>
>>                                                                                         Thanks
>>                                                                                         for
>>                                                                                         the
>>                                                                                         contribution
>>                                                                                         Benny!
>>                                                                                         +1
>>                                                                                         to
>>                                                                                         the
>>                                                                                         confusion
>>                                                                                         the
>>                                                                                         fallback
>>                                                                                         creates.
>>                                                                                         Also
>>                                                                                         just
>>                                                                                         to
>>                                                                                         be
>>                                                                                         clear,
>>                                                                                         at
>>                                                                                         this
>>                                                                                         point
>>                                                                                         and
>>                                                                                         after
>>                                                                                         clarifying
>>                                                                                         the
>>                                                                                         current
>>                                                                                         spec
>>                                                                                         intentions,
>>                                                                                         I
>>                                                                                         am
>>                                                                                         convinced
>>                                                                                         that
>>                                                                                         we
>>                                                                                         should
>>                                                                                         remove
>>                                                                                         the
>>                                                                                         default
>>                                                                                         catalog
>>                                                                                         and
>>                                                                                         default
>>                                                                                         namespace
>>                                                                                         fields
>>                                                                                         altogether.
>>
>>                                                                                         Thanks,
>>                                                                                         Walaa.
>>
>>                                                                                         On
>>                                                                                         Fri,
>>                                                                                         Apr
>>                                                                                         25,
>>                                                                                         2025
>>                                                                                         at
>>                                                                                         5:13 PM
>>                                                                                         Benny
>>                                                                                         Chow
>>                                                                                         <bt...@gmail.com>
>>                                                                                         wrote:
>>
>>                                                                                             I'd
>>                                                                                             like
>>                                                                                             to
>>                                                                                             contribute
>>                                                                                             my
>>                                                                                             opinions
>>                                                                                             on
>>                                                                                             this:
>>
>>                                                                                             -
>>                                                                                             I
>>                                                                                             don't
>>                                                                                             particularly
>>                                                                                             like
>>                                                                                             the
>>                                                                                             current
>>                                                                                             behavior
>>                                                                                             of
>>                                                                                             "default
>>                                                                                             to
>>                                                                                             the
>>                                                                                             view's
>>                                                                                             catalog
>>                                                                                             when
>>                                                                                             default-catalog
>>                                                                                             is
>>                                                                                             not
>>                                                                                             set".
>>                                                                                             Fundamentally,
>>                                                                                             I
>>                                                                                             believe
>>                                                                                             the
>>                                                                                             intent
>>                                                                                             of
>>                                                                                             default-catalog
>>                                                                                             and
>>                                                                                             default-namespace
>>                                                                                             is
>>                                                                                             there
>>                                                                                             to
>>                                                                                             help
>>                                                                                             users
>>                                                                                             write
>>                                                                                             more
>>                                                                                             concise
>>                                                                                             SQL.
>>                                                                                             -
>>                                                                                             spark
>>                                                                                             session
>>                                                                                             catalog
>>                                                                                             is
>>                                                                                             engine
>>                                                                                             specific
>>                                                                                             and
>>                                                                                             I
>>                                                                                             don't
>>                                                                                             think
>>                                                                                             we
>>                                                                                             should
>>                                                                                             design
>>                                                                                             something
>>                                                                                             that
>>                                                                                             says
>>                                                                                             first
>>                                                                                             use
>>                                                                                             this
>>                                                                                             catalog,
>>                                                                                             then
>>                                                                                             that
>>                                                                                             catalog..
>>                                                                                             or
>>                                                                                             that
>>                                                                                             catalog.
>>                                                                                             For
>>                                                                                             example,
>>                                                                                             resolving
>>                                                                                             identifiers
>>                                                                                             using
>>                                                                                             default-catalog
>>                                                                                             ->
>>                                                                                             view's
>>                                                                                             catalog
>>                                                                                             ->
>>                                                                                             session
>>                                                                                             catalog
>>                                                                                             is
>>                                                                                             not
>>                                                                                             good.
>>                                                                                             -
>>                                                                                             We
>>                                                                                             gotta
>>                                                                                             support
>>                                                                                             non-Iceberg
>>                                                                                             tables
>>                                                                                             otherwise
>>                                                                                             I
>>                                                                                             see
>>                                                                                             no
>>                                                                                             value
>>                                                                                             in
>>                                                                                             putting
>>                                                                                             views
>>                                                                                             in
>>                                                                                             the
>>                                                                                             catalog
>>                                                                                             to
>>                                                                                             share
>>                                                                                             with
>>                                                                                             other
>>                                                                                             engines
>>                                                                                             -
>>                                                                                             Interoperability
>>                                                                                             between different engine
>>                                                                                             types
>>                                                                                             is
>>                                                                                             very
>>                                                                                             hard
>>                                                                                             due
>>                                                                                             to
>>                                                                                             dialect issues...
>>                                                                                             so
>>                                                                                             I
>>                                                                                             think
>>                                                                                             we
>>                                                                                             should focus
>>                                                                                             on
>>                                                                                             supporting
>>                                                                                             different
>>                                                                                             clusters
>>                                                                                             of
>>                                                                                             the
>>                                                                                             same
>>                                                                                             engine
>>                                                                                             type
>>                                                                                             on
>>                                                                                             a
>>                                                                                             shared
>>                                                                                             catalog. 
>>                                                                                             For
>>                                                                                             example,
>>                                                                                             AI
>>                                                                                             and
>>                                                                                             BI
>>                                                                                             clusters
>>                                                                                             on
>>                                                                                             Spark
>>                                                                                             sharing
>>                                                                                             the
>>                                                                                             same
>>                                                                                             views
>>                                                                                             in
>>                                                                                             a
>>                                                                                             REST
>>                                                                                             catalog.
>>
>>                                                                                             Coincidentally,
>>                                                                                             I
>>                                                                                             think
>>                                                                                             the
>>                                                                                             ultimate
>>                                                                                             solution
>>                                                                                             is
>>                                                                                             along
>>                                                                                             the
>>                                                                                             lines
>>                                                                                             of
>>                                                                                             something
>>                                                                                             Russell
>>                                                                                             proposed
>>                                                                                             last
>>                                                                                             year:
>>
>>                                                                                             https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>
>>                                                                                             We've
>>                                                                                             been
>>                                                                                             looking
>>                                                                                             at
>>                                                                                             this
>>                                                                                             interoperable
>>                                                                                             identifier
>>                                                                                             problem
>>                                                                                             through
>>                                                                                             the
>>                                                                                             lens
>>                                                                                             of
>>                                                                                             catalog
>>                                                                                             resolution
>>                                                                                             but
>>                                                                                             maybe
>>                                                                                             the
>>                                                                                             right
>>                                                                                             approach
>>                                                                                             is
>>                                                                                             really
>>                                                                                             about
>>                                                                                             templating.
>>
>>                                                                                             I
>>                                                                                             would
>>                                                                                             extend
>>                                                                                             Russell's
>>                                                                                             idea
>>                                                                                             to
>>                                                                                             allow
>>                                                                                             identifiers
>>                                                                                             in
>>                                                                                             a
>>                                                                                             view
>>                                                                                             to
>>                                                                                             span
>>                                                                                             catalogs
>>                                                                                             to
>>                                                                                             support
>>                                                                                             non-Iceberg
>>                                                                                             tables.
>>                                                                                              Also,
>>                                                                                             the
>>                                                                                             default-catalog
>>                                                                                             property
>>                                                                                             could
>>                                                                                             be
>>                                                                                             templated
>>                                                                                             as
>>                                                                                             well.
>>
>>
>>                                                                                             Thoughts?
>>                                                                                             Benny
>>
>>
>>
>>                                                                                             On
>>                                                                                             Fri,
>>                                                                                             Apr
>>                                                                                             25,
>>                                                                                             2025
>>                                                                                             at
>>                                                                                             4:02 PM
>>                                                                                             Walaa
>>                                                                                             Eldin
>>                                                                                             Moustafa
>>                                                                                             <wa...@gmail.com>
>>                                                                                             wrote:
>>
>>                                                                                                 Thanks
>>                                                                                                 Steven!
>>                                                                                                 How
>>                                                                                                 do
>>                                                                                                 you
>>                                                                                                 recommend
>>                                                                                                 making
>>                                                                                                 Spark
>>                                                                                                 implementation
>>                                                                                                 conform
>>                                                                                                 to
>>                                                                                                 the
>>                                                                                                 spec?
>>                                                                                                 Do
>>                                                                                                 we
>>                                                                                                 need
>>                                                                                                 Spark
>>                                                                                                 SQL
>>                                                                                                 extensions
>>                                                                                                 and/or
>>                                                                                                 Spark
>>                                                                                                 catalog
>>                                                                                                 APIs
>>                                                                                                 for
>>                                                                                                 that?
>>
>>                                                                                                 How
>>                                                                                                 do
>>                                                                                                 you
>>                                                                                                 recommend
>>                                                                                                 reconciling
>>                                                                                                 the
>>                                                                                                 inconsistencies
>>                                                                                                 I
>>                                                                                                 shared
>>                                                                                                 regarding
>>                                                                                                 many
>>                                                                                                 resolution
>>                                                                                                 methods
>>                                                                                                 not
>>                                                                                                 consistently
>>                                                                                                 being
>>                                                                                                 followed
>>                                                                                                 in
>>                                                                                                 different
>>                                                                                                 scenarios
>>                                                                                                 (view
>>                                                                                                 vs
>>                                                                                                 child
>>                                                                                                 table
>>                                                                                                 resolution,
>>                                                                                                 query
>>                                                                                                 vs
>>                                                                                                 view
>>                                                                                                 resolution)?
>>                                                                                                 Note
>>                                                                                                 these
>>                                                                                                 occur
>>                                                                                                 when
>>                                                                                                 the
>>                                                                                                 default
>>                                                                                                 catalog
>>                                                                                                 is
>>                                                                                                 set
>>                                                                                                 to
>>                                                                                                 a
>>                                                                                                 non-null
>>                                                                                                 value.
>>                                                                                                 If
>>                                                                                                 it
>>                                                                                                 helps,
>>                                                                                                 I
>>                                                                                                 can
>>                                                                                                 share
>>                                                                                                 concrete
>>                                                                                                 examples.
>>
>>                                                                                                 Thanks,
>>                                                                                                 Walaa.
>>
>>                                                                                                 On
>>                                                                                                 Fri,
>>                                                                                                 Apr
>>                                                                                                 25,
>>                                                                                                 2025
>>                                                                                                 at
>>                                                                                                 3:52 PM
>>                                                                                                 Steven
>>                                                                                                 Wu
>>                                                                                                 <st...@gmail.com>
>>                                                                                                 wrote:
>>
>>                                                                                                     The
>>                                                                                                     core
>>                                                                                                     issue
>>                                                                                                     is
>>                                                                                                     on
>>                                                                                                     the
>>                                                                                                     fall
>>                                                                                                     back
>>                                                                                                     behavior
>>                                                                                                     when
>>                                                                                                     `default-catalog`
>>                                                                                                     is
>>                                                                                                     not
>>                                                                                                     defined.
>>                                                                                                     Current
>>                                                                                                     view
>>                                                                                                     spec
>>                                                                                                     says
>>                                                                                                     the
>>                                                                                                     fallback
>>                                                                                                     should
>>                                                                                                     be
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     where
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     defined.
>>                                                                                                     It
>>                                                                                                     doesn't
>>                                                                                                     really
>>                                                                                                     matter
>>                                                                                                     what
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     is
>>                                                                                                     named
>>                                                                                                     (catalogX)
>>                                                                                                     by
>>                                                                                                     the
>>                                                                                                     read
>>                                                                                                     engine.
>>                                                                                                     -
>>                                                                                                     If
>>                                                                                                     a
>>                                                                                                     view
>>                                                                                                     refers
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     tables
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     catalog,
>>                                                                                                     this
>>                                                                                                     is
>>                                                                                                     a
>>                                                                                                     non-ambiguous
>>                                                                                                     and
>>                                                                                                     reasonable
>>                                                                                                     fallback
>>                                                                                                     behavior.
>>                                                                                                     -
>>                                                                                                     If
>>                                                                                                     a
>>                                                                                                     view
>>                                                                                                     refers
>>                                                                                                     to
>>                                                                                                     tables
>>                                                                                                     from
>>                                                                                                     another
>>                                                                                                     catalog,
>>                                                                                                     catalog
>>                                                                                                     names
>>                                                                                                     should
>>                                                                                                     be
>>                                                                                                     included
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     reference
>>                                                                                                     name
>>                                                                                                     already.
>>                                                                                                     So
>>                                                                                                     no
>>                                                                                                     ambiguity
>>                                                                                                     there
>>                                                                                                     either.
>>
>>                                                                                                     Potential
>>                                                                                                     inconsistent
>>                                                                                                     naming
>>                                                                                                     of
>>                                                                                                     catalog
>>                                                                                                     is
>>                                                                                                     a
>>                                                                                                     separate
>>                                                                                                     problem,
>>                                                                                                     which
>>                                                                                                     Iceberg
>>                                                                                                     view
>>                                                                                                     spec
>>                                                                                                     probably
>>                                                                                                     cannot
>>                                                                                                     solve.
>>                                                                                                     We
>>                                                                                                     can
>>                                                                                                     only
>>                                                                                                     recommend
>>                                                                                                     that
>>                                                                                                     catalog
>>                                                                                                     should
>>                                                                                                     be
>>                                                                                                     named
>>                                                                                                     consistently
>>                                                                                                     across
>>                                                                                                     usage
>>                                                                                                     for
>>                                                                                                     better
>>                                                                                                     interoperability
>>                                                                                                     on
>>                                                                                                     name
>>                                                                                                     references.
>>
>>                                                                                                     This
>>                                                                                                     proposal
>>                                                                                                     is
>>                                                                                                     to
>>                                                                                                     change
>>                                                                                                     the
>>                                                                                                     fallback
>>                                                                                                     behavior
>>                                                                                                     to
>>                                                                                                     engine's
>>                                                                                                     session
>>                                                                                                     default
>>                                                                                                     catalog.
>>                                                                                                     I
>>                                                                                                     am
>>                                                                                                     not
>>                                                                                                     sure
>>                                                                                                     it
>>                                                                                                     is
>>                                                                                                     better
>>                                                                                                     than
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     fallback
>>                                                                                                     behavior.
>>
>>                                                                                                     >
>>                                                                                                     Today’s
>>                                                                                                     Spark
>>                                                                                                     behavior
>>                                                                                                     explicitly
>>                                                                                                     differs
>>                                                                                                     from
>>                                                                                                     this
>>                                                                                                     idea.
>>                                                                                                     Spark
>>                                                                                                     resolves
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     during
>>                                                                                                     view
>>                                                                                                     creation
>>                                                                                                     using
>>                                                                                                     the
>>                                                                                                     session’s
>>                                                                                                     default
>>                                                                                                     catalog,
>>                                                                                                     not
>>                                                                                                     a
>>                                                                                                     supplied
>>                                                                                                     `default-catalog`.
>>
>>                                                                                                     I
>>                                                                                                     would
>>                                                                                                     argue
>>                                                                                                     that
>>                                                                                                     is
>>                                                                                                     a
>>                                                                                                     Spark
>>                                                                                                     implementation
>>                                                                                                     issue
>>                                                                                                     for
>>                                                                                                     not
>>                                                                                                     conforming
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     spec.
>>
>>
>>                                                                                                     On
>>                                                                                                     Fri,
>>                                                                                                     Apr
>>                                                                                                     25,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     1:17 PM
>>                                                                                                     Walaa
>>                                                                                                     Eldin
>>                                                                                                     Moustafa
>>                                                                                                     <wa...@gmail.com>
>>                                                                                                     wrote:
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     Hi
>>                                                                                                     Jan,
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     Thanks
>>                                                                                                     again
>>                                                                                                     for
>>                                                                                                     continuing
>>                                                                                                     the
>>                                                                                                     discussion.
>>                                                                                                     I
>>                                                                                                     want
>>                                                                                                     to
>>                                                                                                     highlight
>>                                                                                                     a
>>                                                                                                     few
>>                                                                                                     fundamental
>>                                                                                                     issues
>>                                                                                                     around
>>                                                                                                     the
>>                                                                                                     interpretation
>>                                                                                                     of
>>                                                                                                     default-catalog:
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     Here
>>                                                                                                     is
>>                                                                                                     the
>>                                                                                                     real
>>                                                                                                     catch:
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     *
>>                                                                                                     default-catalog
>>                                                                                                     cannot
>>                                                                                                     logically
>>                                                                                                     be
>>                                                                                                     defined
>>                                                                                                     at
>>                                                                                                     view
>>                                                                                                     creation
>>                                                                                                     time.
>>                                                                                                     It
>>                                                                                                     would
>>                                                                                                     be
>>                                                                                                     circular:
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     needs
>>                                                                                                     to
>>                                                                                                     exist
>>                                                                                                     before
>>                                                                                                     its
>>                                                                                                     metadata
>>                                                                                                     (and
>>                                                                                                     hence
>>                                                                                                     default-catalog)
>>                                                                                                     can
>>                                                                                                     exist.
>>                                                                                                     This
>>                                                                                                     is
>>                                                                                                     visible
>>                                                                                                     in
>>                                                                                                     Spark’s
>>                                                                                                     implementation,
>>                                                                                                     where
>>                                                                                                     `default-catalog`
>>                                                                                                     is
>>                                                                                                     not
>>                                                                                                     used.
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     *
>>                                                                                                     Introducing
>>                                                                                                     a
>>                                                                                                     creation-time
>>                                                                                                     default-catalog
>>                                                                                                     setting
>>                                                                                                     would
>>                                                                                                     require
>>                                                                                                     extending
>>                                                                                                     SQL
>>                                                                                                     syntax
>>                                                                                                     and
>>                                                                                                     engine
>>                                                                                                     APIs
>>                                                                                                     to
>>                                                                                                     promote
>>                                                                                                     it
>>                                                                                                     to
>>                                                                                                     a
>>                                                                                                     first-class
>>                                                                                                     view
>>                                                                                                     concept.
>>                                                                                                     This
>>                                                                                                     would
>>                                                                                                     be
>>                                                                                                     intrusive,
>>                                                                                                     non-intuitive,
>>                                                                                                     and
>>                                                                                                     realistically
>>                                                                                                     very
>>                                                                                                     difficult
>>                                                                                                     to
>>                                                                                                     standardize
>>                                                                                                     across
>>                                                                                                     engines.
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     *
>>                                                                                                     Today’s
>>                                                                                                     Spark
>>                                                                                                     behavior
>>                                                                                                     explicitly
>>                                                                                                     differs
>>                                                                                                     from
>>                                                                                                     this
>>                                                                                                     idea.
>>                                                                                                     Spark
>>                                                                                                     resolves
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     during
>>                                                                                                     view
>>                                                                                                     creation
>>                                                                                                     using
>>                                                                                                     the
>>                                                                                                     session’s
>>                                                                                                     default
>>                                                                                                     catalog,
>>                                                                                                     not
>>                                                                                                     a
>>                                                                                                     supplied
>>                                                                                                     `default-catalog`.
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     *
>>                                                                                                     Hypothetically
>>                                                                                                     even
>>                                                                                                     if
>>                                                                                                     we
>>                                                                                                     patched
>>                                                                                                     in
>>                                                                                                     a
>>                                                                                                     creation-time
>>                                                                                                     default-catalog,
>>                                                                                                     it
>>                                                                                                     would
>>                                                                                                     create
>>                                                                                                     an
>>                                                                                                     inconsistent
>>                                                                                                     binding
>>                                                                                                     model
>>                                                                                                     between
>>                                                                                                     tables
>>                                                                                                     vs
>>                                                                                                     views
>>                                                                                                     (early
>>                                                                                                     vs
>>                                                                                                     late),
>>                                                                                                     and
>>                                                                                                     between
>>                                                                                                     tables
>>                                                                                                     in
>>                                                                                                     views
>>                                                                                                     and
>>                                                                                                     in
>>                                                                                                     queries
>>                                                                                                     (again
>>                                                                                                     early
>>                                                                                                     vs
>>                                                                                                     late).
>>                                                                                                     For
>>                                                                                                     example,
>>                                                                                                     views
>>                                                                                                     and
>>                                                                                                     tables
>>                                                                                                     in
>>                                                                                                     queries
>>                                                                                                     can
>>                                                                                                     withstand
>>                                                                                                     default
>>                                                                                                     catalog
>>                                                                                                     renames,
>>                                                                                                     but
>>                                                                                                     tables
>>                                                                                                     cannot
>>                                                                                                     when
>>                                                                                                     they
>>                                                                                                     are
>>                                                                                                     used
>>                                                                                                     inside
>>                                                                                                     views
>>                                                                                                     --
>>                                                                                                     it
>>                                                                                                     even
>>                                                                                                     applies
>>                                                                                                     to
>>                                                                                                     views
>>                                                                                                     inside
>>                                                                                                     views,
>>                                                                                                     which
>>                                                                                                     makes
>>                                                                                                     this
>>                                                                                                     very
>>                                                                                                     hard
>>                                                                                                     to
>>                                                                                                     reason
>>                                                                                                     about
>>                                                                                                     considering
>>                                                                                                     nesting.
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     Thanks,
>>                                                                                                     >
>>                                                                                                     Walaa
>>                                                                                                     >
>>                                                                                                     >
>>                                                                                                     On
>>                                                                                                     Fri,
>>                                                                                                     Apr
>>                                                                                                     25,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     7:00 AM
>>                                                                                                     Jan
>>                                                                                                     Kaul
>>                                                                                                     <ja...@mailbox.org.invalid>
>>                                                                                                     <ma...@mailbox.org.invalid>
>>                                                                                                     wrote:
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     @Walaa:
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     I
>>                                                                                                     would
>>                                                                                                     argue
>>                                                                                                     that
>>                                                                                                     when
>>                                                                                                     you
>>                                                                                                     run
>>                                                                                                     a
>>                                                                                                     CREATE
>>                                                                                                     VIEW
>>                                                                                                     statement
>>                                                                                                     the
>>                                                                                                     query
>>                                                                                                     engine
>>                                                                                                     knowns
>>                                                                                                     which
>>                                                                                                     catalog
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     being
>>                                                                                                     created
>>                                                                                                     in.
>>                                                                                                     So
>>                                                                                                     even
>>                                                                                                     though
>>                                                                                                     we
>>                                                                                                     typically
>>                                                                                                     use
>>                                                                                                     late
>>                                                                                                     binding
>>                                                                                                     to
>>                                                                                                     resolve
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     catalog
>>                                                                                                     at
>>                                                                                                     query
>>                                                                                                     time,
>>                                                                                                     it
>>                                                                                                     can
>>                                                                                                     also
>>                                                                                                     be
>>                                                                                                     used
>>                                                                                                     at
>>                                                                                                     creation
>>                                                                                                     time.
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     The
>>                                                                                                     query
>>                                                                                                     engine
>>                                                                                                     would
>>                                                                                                     need
>>                                                                                                     to
>>                                                                                                     keep
>>                                                                                                     track
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     "view
>>                                                                                                     catalog"
>>                                                                                                     where
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     going
>>                                                                                                     to
>>                                                                                                     be
>>                                                                                                     created
>>                                                                                                     in.
>>                                                                                                     It
>>                                                                                                     can
>>                                                                                                     use
>>                                                                                                     that
>>                                                                                                     catalog
>>                                                                                                     to
>>                                                                                                     resolve
>>                                                                                                     partial
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     if
>>                                                                                                     "default-catalog"
>>                                                                                                     is
>>                                                                                                     not
>>                                                                                                     set.
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     It
>>                                                                                                     can
>>                                                                                                     lead
>>                                                                                                     to
>>                                                                                                     some
>>                                                                                                     unintuitive
>>                                                                                                     behavior,
>>                                                                                                     where
>>                                                                                                     partial
>>                                                                                                     identifiers
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     query
>>                                                                                                     resolve
>>                                                                                                     to
>>                                                                                                     a
>>                                                                                                     different
>>                                                                                                     catalog
>>                                                                                                     compared
>>                                                                                                     to
>>                                                                                                     using
>>                                                                                                     them
>>                                                                                                     outside
>>                                                                                                     of
>>                                                                                                     a
>>                                                                                                     view.
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     CREATE
>>                                                                                                     VIEW
>>                                                                                                     catalogA.sales.monthly_orders
>>                                                                                                     AS
>>                                                                                                     SELECT
>>                                                                                                     *
>>                                                                                                     from
>>                                                                                                     sales.orders;
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     If
>>                                                                                                     the
>>                                                                                                     session
>>                                                                                                     default
>>                                                                                                     catalog
>>                                                                                                     is
>>                                                                                                     not
>>                                                                                                     "catalogA",
>>                                                                                                     the
>>                                                                                                     "sales.orders"
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     query
>>                                                                                                     would
>>                                                                                                     not
>>                                                                                                     be
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     as
>>                                                                                                     just
>>                                                                                                     referencing
>>                                                                                                     "sales.orders"
>>                                                                                                     in
>>                                                                                                     a
>>                                                                                                     normal
>>                                                                                                     SQL
>>                                                                                                     statement.
>>                                                                                                     This
>>                                                                                                     is
>>                                                                                                     because
>>                                                                                                     without
>>                                                                                                     a
>>                                                                                                     "default-catalog",
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     name
>>                                                                                                     of
>>                                                                                                     "sales.orders"
>>                                                                                                     would
>>                                                                                                     default
>>                                                                                                     to
>>                                                                                                     "catalogA",
>>                                                                                                     which
>>                                                                                                     is
>>                                                                                                     the
>>                                                                                                     view's
>>                                                                                                     catalog.
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     Thanks,
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     Jan
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     On
>>                                                                                                     4/25/25
>>                                                                                                     04:05,
>>                                                                                                     Manu
>>                                                                                                     Zhang
>>                                                                                                     wrote:
>>                                                                                                     >>>
>>                                                                                                     >>>
>>                                                                                                     For
>>                                                                                                     example,
>>                                                                                                     if
>>                                                                                                     we
>>                                                                                                     want
>>                                                                                                     to
>>                                                                                                     validate
>>                                                                                                     that
>>                                                                                                     the
>>                                                                                                     tables
>>                                                                                                     referenced
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     exist,
>>                                                                                                     how
>>                                                                                                     can
>>                                                                                                     we
>>                                                                                                     do
>>                                                                                                     that
>>                                                                                                     when
>>                                                                                                     default-catalog
>>                                                                                                     isn't
>>                                                                                                     defined,
>>                                                                                                     since
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     hasn't
>>                                                                                                     been
>>                                                                                                     created
>>                                                                                                     or
>>                                                                                                     loaded
>>                                                                                                     yet?
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     I
>>                                                                                                     don't
>>                                                                                                     think
>>                                                                                                     this
>>                                                                                                     is
>>                                                                                                     related
>>                                                                                                     to
>>                                                                                                     view
>>                                                                                                     spec.
>>                                                                                                     How
>>                                                                                                     do
>>                                                                                                     we
>>                                                                                                     validate
>>                                                                                                     that
>>                                                                                                     a
>>                                                                                                     table
>>                                                                                                     exists
>>                                                                                                     without
>>                                                                                                     a
>>                                                                                                     default
>>                                                                                                     catalog,
>>                                                                                                     or
>>                                                                                                     do
>>                                                                                                     we
>>                                                                                                     always
>>                                                                                                     use
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     session
>>                                                                                                     catalog?
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     Thanks,
>>                                                                                                     >>
>>                                                                                                     Manu
>>                                                                                                     >>
>>                                                                                                     >>
>>                                                                                                     On
>>                                                                                                     Fri,
>>                                                                                                     Apr
>>                                                                                                     25,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     5:59 AM
>>                                                                                                     Walaa
>>                                                                                                     Eldin
>>                                                                                                     Moustafa
>>                                                                                                     <wa...@gmail.com>
>>                                                                                                     wrote:
>>                                                                                                     >>>
>>                                                                                                     >>>
>>                                                                                                     Hi
>>                                                                                                     Jan,
>>                                                                                                     >>>
>>                                                                                                     >>>
>>                                                                                                     I
>>                                                                                                     think
>>                                                                                                     we
>>                                                                                                     still
>>                                                                                                     share
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     understanding.
>>                                                                                                     Just
>>                                                                                                     to
>>                                                                                                     clarify:
>>                                                                                                     when
>>                                                                                                     I
>>                                                                                                     referred
>>                                                                                                     to
>>                                                                                                     late
>>                                                                                                     binding
>>                                                                                                     as
>>                                                                                                     “similar”
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     proposal,
>>                                                                                                     I
>>                                                                                                     was
>>                                                                                                     acknowledging
>>                                                                                                     the
>>                                                                                                     distinction
>>                                                                                                     between
>>                                                                                                     view-level
>>                                                                                                     and
>>                                                                                                     table-level
>>                                                                                                     resolution.
>>                                                                                                     But
>>                                                                                                     as
>>                                                                                                     you
>>                                                                                                     noted,
>>                                                                                                     both
>>                                                                                                     follow
>>                                                                                                     a
>>                                                                                                     late
>>                                                                                                     binding
>>                                                                                                     model.
>>                                                                                                     >>>
>>                                                                                                     >>>
>>                                                                                                     That
>>                                                                                                     said,
>>                                                                                                     this
>>                                                                                                     still
>>                                                                                                     raises
>>                                                                                                     an
>>                                                                                                     interesting
>>                                                                                                     question
>>                                                                                                     and
>>                                                                                                     a
>>                                                                                                     potential
>>                                                                                                     gap:
>>                                                                                                     if
>>                                                                                                     default-catalog
>>                                                                                                     is
>>                                                                                                     only
>>                                                                                                     defined
>>                                                                                                     at
>>                                                                                                     query
>>                                                                                                     time,
>>                                                                                                     how
>>                                                                                                     should
>>                                                                                                     resolution
>>                                                                                                     work
>>                                                                                                     during
>>                                                                                                     view
>>                                                                                                     creation?
>>                                                                                                     For
>>                                                                                                     example,
>>                                                                                                     if
>>                                                                                                     we
>>                                                                                                     want
>>                                                                                                     to
>>                                                                                                     validate
>>                                                                                                     that
>>                                                                                                     the
>>                                                                                                     tables
>>                                                                                                     referenced
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     exist,
>>                                                                                                     how
>>                                                                                                     can
>>                                                                                                     we
>>                                                                                                     do
>>                                                                                                     that
>>                                                                                                     when
>>                                                                                                     default-catalog
>>                                                                                                     isn't
>>                                                                                                     defined,
>>                                                                                                     since
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     hasn't
>>                                                                                                     been
>>                                                                                                     created
>>                                                                                                     or
>>                                                                                                     loaded
>>                                                                                                     yet?
>>                                                                                                     >>>
>>                                                                                                     >>>
>>                                                                                                     Thanks,
>>                                                                                                     >>>
>>                                                                                                     Walaa.
>>                                                                                                     >>>
>>                                                                                                     >>>
>>                                                                                                     On
>>                                                                                                     Thu,
>>                                                                                                     Apr
>>                                                                                                     24,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     7:02 AM
>>                                                                                                     Jan
>>                                                                                                     Kaul
>>                                                                                                     <ja...@mailbox.org.invalid>
>>                                                                                                     <ma...@mailbox.org.invalid>
>>                                                                                                     wrote:
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     Yes,
>>                                                                                                     I
>>                                                                                                     have
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     understanding.
>>                                                                                                     The
>>                                                                                                     view
>>                                                                                                     catalog
>>                                                                                                     is
>>                                                                                                     resolved
>>                                                                                                     at
>>                                                                                                     query
>>                                                                                                     time.
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     As
>>                                                                                                     you
>>                                                                                                     mentioned
>>                                                                                                     before,
>>                                                                                                     it's
>>                                                                                                     good
>>                                                                                                     to
>>                                                                                                     distinguish
>>                                                                                                     between
>>                                                                                                     the
>>                                                                                                     physical
>>                                                                                                     catalog
>>                                                                                                     and
>>                                                                                                     it's
>>                                                                                                     reference
>>                                                                                                     used
>>                                                                                                     in
>>                                                                                                     SQL
>>                                                                                                     statements.
>>                                                                                                     The
>>                                                                                                     important
>>                                                                                                     part
>>                                                                                                     is
>>                                                                                                     that
>>                                                                                                     the
>>                                                                                                     physical
>>                                                                                                     catalog
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     and
>>                                                                                                     the
>>                                                                                                     tables
>>                                                                                                     referenced
>>                                                                                                     in
>>                                                                                                     it's
>>                                                                                                     definition
>>                                                                                                     stay
>>                                                                                                     consistent.
>>                                                                                                     You
>>                                                                                                     could
>>                                                                                                     create
>>                                                                                                     a
>>                                                                                                     view
>>                                                                                                     in
>>                                                                                                     a
>>                                                                                                     given
>>                                                                                                     physical
>>                                                                                                     catalog
>>                                                                                                     by
>>                                                                                                     referring
>>                                                                                                     to
>>                                                                                                     it
>>                                                                                                     as
>>                                                                                                     "catalogA",
>>                                                                                                     as
>>                                                                                                     in
>>                                                                                                     your
>>                                                                                                     first
>>                                                                                                     point.
>>                                                                                                     If
>>                                                                                                     you
>>                                                                                                     then,
>>                                                                                                     given
>>                                                                                                     a
>>                                                                                                     different
>>                                                                                                     setup,
>>                                                                                                     refer
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     physical
>>                                                                                                     catalog
>>                                                                                                     as
>>                                                                                                     "catalogB"
>>                                                                                                     in
>>                                                                                                     another
>>                                                                                                     session/environment,
>>                                                                                                     the
>>                                                                                                     behavior
>>                                                                                                     should
>>                                                                                                     still
>>                                                                                                     work.
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     I
>>                                                                                                     would
>>                                                                                                     however
>>                                                                                                     rephrase
>>                                                                                                     your
>>                                                                                                     last
>>                                                                                                     point.
>>                                                                                                     Late
>>                                                                                                     binding
>>                                                                                                     applies
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     catalog
>>                                                                                                     name
>>                                                                                                     and
>>                                                                                                     by
>>                                                                                                     extension
>>                                                                                                     to
>>                                                                                                     all
>>                                                                                                     partial
>>                                                                                                     table
>>                                                                                                     references
>>                                                                                                     when
>>                                                                                                     no
>>                                                                                                     "default-catalog"
>>                                                                                                     is
>>                                                                                                     present.
>>                                                                                                     Resolving
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     catalog
>>                                                                                                     name
>>                                                                                                     at
>>                                                                                                     query
>>                                                                                                     time
>>                                                                                                     is
>>                                                                                                     not
>>                                                                                                     opposed
>>                                                                                                     to
>>                                                                                                     storing
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     metadata
>>                                                                                                     in
>>                                                                                                     a
>>                                                                                                     catalog.
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     Or
>>                                                                                                     maybe
>>                                                                                                     I
>>                                                                                                     don't
>>                                                                                                     entirely
>>                                                                                                     understand
>>                                                                                                     what
>>                                                                                                     you
>>                                                                                                     mean.
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     Thanks
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     Jan
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     On
>>                                                                                                     4/24/25
>>                                                                                                     00:32,
>>                                                                                                     Walaa
>>                                                                                                     Eldin
>>                                                                                                     Moustafa
>>                                                                                                     wrote:
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     Hi
>>                                                                                                     Jan,
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     >
>>                                                                                                     The
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     executed
>>                                                                                                     when
>>                                                                                                     it's
>>                                                                                                     being
>>                                                                                                     referenced
>>                                                                                                     in
>>                                                                                                     a
>>                                                                                                     SQL
>>                                                                                                     statement.
>>                                                                                                     That
>>                                                                                                     statement
>>                                                                                                     contains
>>                                                                                                     the
>>                                                                                                     information
>>                                                                                                     for
>>                                                                                                     the
>>                                                                                                     query
>>                                                                                                     engine
>>                                                                                                     to
>>                                                                                                     resolve
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     view.
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     If
>>                                                                                                     I’m
>>                                                                                                     understanding
>>                                                                                                     correctly,
>>                                                                                                     that
>>                                                                                                     means:
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     *
>>                                                                                                     If
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     queried
>>                                                                                                     as
>>                                                                                                     SELECT
>>                                                                                                     *
>>                                                                                                     FROM
>>                                                                                                     catalogA.namespace.view,
>>                                                                                                     then
>>                                                                                                     catalogA
>>                                                                                                     is
>>                                                                                                     considered
>>                                                                                                     the
>>                                                                                                     view’s
>>                                                                                                     catalog.
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     *
>>                                                                                                     If
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     later
>>                                                                                                     queried
>>                                                                                                     as
>>                                                                                                     SELECT
>>                                                                                                     *
>>                                                                                                     FROM
>>                                                                                                     catalogB.namespace.view
>>                                                                                                     (after
>>                                                                                                     renaming
>>                                                                                                     catalogA
>>                                                                                                     to
>>                                                                                                     catalogB,
>>                                                                                                     and
>>                                                                                                     keeping
>>                                                                                                     everything
>>                                                                                                     else
>>                                                                                                     the
>>                                                                                                     same),
>>                                                                                                     then
>>                                                                                                     catalogB
>>                                                                                                     becomes
>>                                                                                                     the
>>                                                                                                     view’s
>>                                                                                                     catalog.
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     Is
>>                                                                                                     that
>>                                                                                                     interpretation
>>                                                                                                     correct?
>>                                                                                                     If
>>                                                                                                     so,
>>                                                                                                     it
>>                                                                                                     sounds
>>                                                                                                     to
>>                                                                                                     me
>>                                                                                                     like
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     is
>>                                                                                                     resolved
>>                                                                                                     at
>>                                                                                                     query
>>                                                                                                     time,
>>                                                                                                     based
>>                                                                                                     on
>>                                                                                                     how
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     referenced,
>>                                                                                                     not
>>                                                                                                     from
>>                                                                                                     any
>>                                                                                                     stored
>>                                                                                                     metadata.
>>                                                                                                     That
>>                                                                                                     would
>>                                                                                                     imply
>>                                                                                                     some
>>                                                                                                     sort
>>                                                                                                     of
>>                                                                                                     a
>>                                                                                                     late
>>                                                                                                     binding
>>                                                                                                     behavior
>>                                                                                                     (similar
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     proposal),
>>                                                                                                     as
>>                                                                                                     opposed
>>                                                                                                     to
>>                                                                                                     using
>>                                                                                                     some
>>                                                                                                     catalog
>>                                                                                                     that
>>                                                                                                     "stores"
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     definition.
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     Thanks,
>>                                                                                                     >>>>
>>                                                                                                     Walaa
>>                                                                                                     >>>>
>>                                                                                                     >>>>
>>                                                                                                     On
>>                                                                                                     Tue,
>>                                                                                                     Apr
>>                                                                                                     22,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     11:01 AM
>>                                                                                                     Jan
>>                                                                                                     Kaul
>>                                                                                                     <ja...@mailbox.org.invalid>
>>                                                                                                     <ma...@mailbox.org.invalid>
>>                                                                                                     wrote:
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     Hi
>>                                                                                                     Walaa,
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     Thanks
>>                                                                                                     for
>>                                                                                                     clarifying
>>                                                                                                     the
>>                                                                                                     aspects
>>                                                                                                     of
>>                                                                                                     non-determinism.
>>                                                                                                     Let
>>                                                                                                     me
>>                                                                                                     try
>>                                                                                                     to
>>                                                                                                     address
>>                                                                                                     your
>>                                                                                                     questions.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     1.
>>                                                                                                     This
>>                                                                                                     is
>>                                                                                                     my
>>                                                                                                     interpretation
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     spec:
>>                                                                                                     The
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     executed
>>                                                                                                     when
>>                                                                                                     it's
>>                                                                                                     being
>>                                                                                                     referenced
>>                                                                                                     in
>>                                                                                                     a
>>                                                                                                     SQL
>>                                                                                                     statement.
>>                                                                                                     That
>>                                                                                                     statement
>>                                                                                                     contains
>>                                                                                                     the
>>                                                                                                     information
>>                                                                                                     for
>>                                                                                                     the
>>                                                                                                     query
>>                                                                                                     engine
>>                                                                                                     to
>>                                                                                                     resolve
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     view.
>>                                                                                                     The
>>                                                                                                     query
>>                                                                                                     engine
>>                                                                                                     then
>>                                                                                                     uses
>>                                                                                                     that
>>                                                                                                     information
>>                                                                                                     to
>>                                                                                                     fetch
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     metadata
>>                                                                                                     from
>>                                                                                                     the
>>                                                                                                     catalog.
>>                                                                                                     It
>>                                                                                                     also
>>                                                                                                     needs
>>                                                                                                     to
>>                                                                                                     temporarily
>>                                                                                                     keep
>>                                                                                                     track
>>                                                                                                     of
>>                                                                                                     which
>>                                                                                                     catalog
>>                                                                                                     it
>>                                                                                                     used
>>                                                                                                     to
>>                                                                                                     fetch
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     metadata.
>>                                                                                                     It
>>                                                                                                     can
>>                                                                                                     then
>>                                                                                                     use
>>                                                                                                     that
>>                                                                                                     information
>>                                                                                                     to
>>                                                                                                     resolve
>>                                                                                                     the
>>                                                                                                     table
>>                                                                                                     references
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     views
>>                                                                                                     SQL
>>                                                                                                     definition
>>                                                                                                     in
>>                                                                                                     case
>>                                                                                                     no
>>                                                                                                     default
>>                                                                                                     catalog
>>                                                                                                     is
>>                                                                                                     specified.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     2.
>>                                                                                                     The
>>                                                                                                     important
>>                                                                                                     part
>>                                                                                                     is
>>                                                                                                     that
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     can
>>                                                                                                     be
>>                                                                                                     referenced
>>                                                                                                     at
>>                                                                                                     execution
>>                                                                                                     time.
>>                                                                                                     As
>>                                                                                                     long
>>                                                                                                     as
>>                                                                                                     that's
>>                                                                                                     the
>>                                                                                                     case
>>                                                                                                     I
>>                                                                                                     would
>>                                                                                                     assume
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     can
>>                                                                                                     be
>>                                                                                                     created
>>                                                                                                     in
>>                                                                                                     any
>>                                                                                                     catalog.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     I
>>                                                                                                     think
>>                                                                                                     your
>>                                                                                                     point
>>                                                                                                     is
>>                                                                                                     really
>>                                                                                                     valuable
>>                                                                                                     because
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     specification
>>                                                                                                     can
>>                                                                                                     lead
>>                                                                                                     to
>>                                                                                                     some
>>                                                                                                     unintuitive
>>                                                                                                     behavior.
>>                                                                                                     For
>>                                                                                                     example
>>                                                                                                     for
>>                                                                                                     the
>>                                                                                                     following
>>                                                                                                     statement:
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     CREATE
>>                                                                                                     VIEW
>>                                                                                                     catalogA.sales.monthly_orders
>>                                                                                                     AS
>>                                                                                                     SELECT
>>                                                                                                     *
>>                                                                                                     from
>>                                                                                                     sales.orders;
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     If
>>                                                                                                     the
>>                                                                                                     session
>>                                                                                                     default
>>                                                                                                     catalog
>>                                                                                                     is
>>                                                                                                     not
>>                                                                                                     "catalogA",
>>                                                                                                     the
>>                                                                                                     "sales.orders"
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     query
>>                                                                                                     would
>>                                                                                                     not
>>                                                                                                     be
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     as
>>                                                                                                     just
>>                                                                                                     referencing
>>                                                                                                     "sales.orders"
>>                                                                                                     in
>>                                                                                                     a
>>                                                                                                     normal
>>                                                                                                     SQL
>>                                                                                                     statement.
>>                                                                                                     This
>>                                                                                                     is
>>                                                                                                     because
>>                                                                                                     without
>>                                                                                                     a
>>                                                                                                     "default-catalog",
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     name
>>                                                                                                     of
>>                                                                                                     "sales.orders"
>>                                                                                                     would
>>                                                                                                     default
>>                                                                                                     to
>>                                                                                                     "catalogA".
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     However,
>>                                                                                                     I
>>                                                                                                     like
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     design
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     spec,
>>                                                                                                     because
>>                                                                                                     it
>>                                                                                                     has
>>                                                                                                     the
>>                                                                                                     "closure"
>>                                                                                                     property.
>>                                                                                                     Because
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     fact
>>                                                                                                     that
>>                                                                                                     the
>>                                                                                                     "view
>>                                                                                                     catalog"
>>                                                                                                     has
>>                                                                                                     to
>>                                                                                                     be
>>                                                                                                     known
>>                                                                                                     when
>>                                                                                                     executing
>>                                                                                                     a
>>                                                                                                     view,
>>                                                                                                     all
>>                                                                                                     the
>>                                                                                                     information
>>                                                                                                     required
>>                                                                                                     to
>>                                                                                                     resolve
>>                                                                                                     the
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     is
>>                                                                                                     contained
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     metadata
>>                                                                                                     (and
>>                                                                                                     the
>>                                                                                                     "view
>>                                                                                                     catalog").
>>                                                                                                     I
>>                                                                                                     think
>>                                                                                                     that
>>                                                                                                     if
>>                                                                                                     you
>>                                                                                                     make
>>                                                                                                     the
>>                                                                                                     identifier
>>                                                                                                     resolution
>>                                                                                                     dependent
>>                                                                                                     on
>>                                                                                                     external
>>                                                                                                     parameters,
>>                                                                                                     it
>>                                                                                                     hinders
>>                                                                                                     portability.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     Thanks,
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     Jan
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     On
>>                                                                                                     4/22/25
>>                                                                                                     18:36,
>>                                                                                                     Walaa
>>                                                                                                     Eldin
>>                                                                                                     Moustafa
>>                                                                                                     wrote:
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     Hi
>>                                                                                                     Jan,
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     Thanks
>>                                                                                                     for
>>                                                                                                     the
>>                                                                                                     thoughtful
>>                                                                                                     feedback.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     I
>>                                                                                                     think
>>                                                                                                     it’s
>>                                                                                                     important
>>                                                                                                     we
>>                                                                                                     clarify
>>                                                                                                     a
>>                                                                                                     key
>>                                                                                                     point
>>                                                                                                     before
>>                                                                                                     going
>>                                                                                                     deeper:
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     Non-determinism
>>                                                                                                     is
>>                                                                                                     not
>>                                                                                                     caused
>>                                                                                                     by
>>                                                                                                     session
>>                                                                                                     fallback
>>                                                                                                     behavior—it’s
>>                                                                                                     a
>>                                                                                                     fundamental
>>                                                                                                     limitation
>>                                                                                                     of
>>                                                                                                     using
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     alone,
>>                                                                                                     regardless
>>                                                                                                     of
>>                                                                                                     whether
>>                                                                                                     we
>>                                                                                                     use
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     rule,
>>                                                                                                     the
>>                                                                                                     proposed
>>                                                                                                     fallback
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     session’s
>>                                                                                                     default
>>                                                                                                     catalog,
>>                                                                                                     or
>>                                                                                                     even
>>                                                                                                     early
>>                                                                                                     vs.
>>                                                                                                     late
>>                                                                                                     binding.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     The
>>                                                                                                     same
>>                                                                                                     fully
>>                                                                                                     qualified
>>                                                                                                     identifier
>>                                                                                                     (e.g.,
>>                                                                                                     catalogA.namespace.table)
>>                                                                                                     can
>>                                                                                                     resolve
>>                                                                                                     to
>>                                                                                                     different
>>                                                                                                     objects
>>                                                                                                     depending
>>                                                                                                     solely
>>                                                                                                     on
>>                                                                                                     engine-specific
>>                                                                                                     routing
>>                                                                                                     logic
>>                                                                                                     or
>>                                                                                                     catalog
>>                                                                                                     aliases.
>>                                                                                                     So
>>                                                                                                     determinism
>>                                                                                                     isn’t
>>                                                                                                     guaranteed
>>                                                                                                     just
>>                                                                                                     because
>>                                                                                                     an
>>                                                                                                     identifier
>>                                                                                                     is
>>                                                                                                     "fully
>>                                                                                                     qualified."
>>                                                                                                     The
>>                                                                                                     only
>>                                                                                                     reliable
>>                                                                                                     anchor
>>                                                                                                     for
>>                                                                                                     identity
>>                                                                                                     is
>>                                                                                                     the
>>                                                                                                     UUID.
>>                                                                                                     That’s
>>                                                                                                     why
>>                                                                                                     the
>>                                                                                                     proposed
>>                                                                                                     use
>>                                                                                                     of
>>                                                                                                     UUIDs
>>                                                                                                     is
>>                                                                                                     not
>>                                                                                                     just
>>                                                                                                     a
>>                                                                                                     hardening
>>                                                                                                     strategy.
>>                                                                                                     It’s
>>                                                                                                     the
>>                                                                                                     actual
>>                                                                                                     fix
>>                                                                                                     for
>>                                                                                                     correctness.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     To
>>                                                                                                     move
>>                                                                                                     the
>>                                                                                                     conversation
>>                                                                                                     forward,
>>                                                                                                     could
>>                                                                                                     you
>>                                                                                                     help
>>                                                                                                     clarify
>>                                                                                                     two
>>                                                                                                     things
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     context
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     spec:
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     *
>>                                                                                                     Where
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     metadata
>>                                                                                                     is
>>                                                                                                     the
>>                                                                                                     “view
>>                                                                                                     catalog”
>>                                                                                                     stored,
>>                                                                                                     so
>>                                                                                                     that
>>                                                                                                     an
>>                                                                                                     engine
>>                                                                                                     knows
>>                                                                                                     to
>>                                                                                                     fall
>>                                                                                                     back
>>                                                                                                     to
>>                                                                                                     it
>>                                                                                                     if
>>                                                                                                     default-catalog
>>                                                                                                     is
>>                                                                                                     null?
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     *
>>                                                                                                     Are
>>                                                                                                     we
>>                                                                                                     even
>>                                                                                                     allowed
>>                                                                                                     to
>>                                                                                                     create
>>                                                                                                     views
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     session's
>>                                                                                                     default
>>                                                                                                     catalog
>>                                                                                                     (i.e.,
>>                                                                                                     without
>>                                                                                                     specifying
>>                                                                                                     a
>>                                                                                                     catalog)
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     Iceberg
>>                                                                                                     spec?
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     These
>>                                                                                                     questions
>>                                                                                                     are
>>                                                                                                     important
>>                                                                                                     because
>>                                                                                                     if
>>                                                                                                     we
>>                                                                                                     can’t
>>                                                                                                     unambiguously
>>                                                                                                     recover
>>                                                                                                     the
>>                                                                                                     "view
>>                                                                                                     catalog"
>>                                                                                                     from
>>                                                                                                     metadata,
>>                                                                                                     then
>>                                                                                                     defaulting
>>                                                                                                     to
>>                                                                                                     it
>>                                                                                                     is
>>                                                                                                     problematic.
>>                                                                                                     And
>>                                                                                                     if
>>                                                                                                     views
>>                                                                                                     can't
>>                                                                                                     be
>>                                                                                                     created
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     default
>>                                                                                                     catalog,
>>                                                                                                     then
>>                                                                                                     the
>>                                                                                                     fallback
>>                                                                                                     rule
>>                                                                                                     doesn’t
>>                                                                                                     generalize.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     Thanks,
>>                                                                                                     >>>>>
>>                                                                                                     Walaa.
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     >>>>>
>>                                                                                                     On
>>                                                                                                     Tue,
>>                                                                                                     Apr
>>                                                                                                     22,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     3:14 AM
>>                                                                                                     Jan
>>                                                                                                     Kaul
>>                                                                                                     <ja...@mailbox.org.invalid>
>>                                                                                                     <ma...@mailbox.org.invalid>
>>                                                                                                     wrote:
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Hi
>>                                                                                                     Walaa,
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     thank
>>                                                                                                     you
>>                                                                                                     for
>>                                                                                                     your
>>                                                                                                     proposal.
>>                                                                                                     If
>>                                                                                                     I
>>                                                                                                     understood
>>                                                                                                     correctly,
>>                                                                                                     you
>>                                                                                                     proposal
>>                                                                                                     is
>>                                                                                                     composed
>>                                                                                                     of
>>                                                                                                     three
>>                                                                                                     parts:
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     -
>>                                                                                                     session
>>                                                                                                     default
>>                                                                                                     catalog
>>                                                                                                     as
>>                                                                                                     fallback
>>                                                                                                     for
>>                                                                                                     "default-catalog"
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     -
>>                                                                                                     session
>>                                                                                                     default
>>                                                                                                     namespace
>>                                                                                                     as
>>                                                                                                     fallback
>>                                                                                                     for
>>                                                                                                     "default-namepace"
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     -
>>                                                                                                     Late
>>                                                                                                     binding
>>                                                                                                     +
>>                                                                                                     UUID
>>                                                                                                     validation
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     I
>>                                                                                                     have
>>                                                                                                     some
>>                                                                                                     comments
>>                                                                                                     regarding
>>                                                                                                     these
>>                                                                                                     points.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     1.
>>                                                                                                     Session
>>                                                                                                     default
>>                                                                                                     catalog
>>                                                                                                     as
>>                                                                                                     fallback
>>                                                                                                     for
>>                                                                                                     "default-catalog"
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Introducing
>>                                                                                                     a
>>                                                                                                     behavior
>>                                                                                                     that
>>                                                                                                     depends
>>                                                                                                     on
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     session
>>                                                                                                     setup
>>                                                                                                     is
>>                                                                                                     in
>>                                                                                                     my
>>                                                                                                     opinion
>>                                                                                                     the
>>                                                                                                     definition
>>                                                                                                     of
>>                                                                                                     "non-determinism".
>>                                                                                                     You
>>                                                                                                     could
>>                                                                                                     be
>>                                                                                                     running
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     query-engine
>>                                                                                                     and
>>                                                                                                     catalog-setup
>>                                                                                                     on
>>                                                                                                     different
>>                                                                                                     days,
>>                                                                                                     with
>>                                                                                                     different
>>                                                                                                     default
>>                                                                                                     session
>>                                                                                                     catalogs
>>                                                                                                     (which
>>                                                                                                     is
>>                                                                                                     rather
>>                                                                                                     common),
>>                                                                                                     and
>>                                                                                                     would
>>                                                                                                     be
>>                                                                                                     getting
>>                                                                                                     different
>>                                                                                                     results.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Whereas
>>                                                                                                     with
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     behavior,
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     always
>>                                                                                                     produces
>>                                                                                                     the
>>                                                                                                     same
>>                                                                                                     results.
>>                                                                                                     The
>>                                                                                                     current
>>                                                                                                     behavior
>>                                                                                                     has
>>                                                                                                     some
>>                                                                                                     rough
>>                                                                                                     edges
>>                                                                                                     in
>>                                                                                                     very
>>                                                                                                     niche
>>                                                                                                     use
>>                                                                                                     cases
>>                                                                                                     but
>>                                                                                                     I
>>                                                                                                     think
>>                                                                                                     is
>>                                                                                                     solid
>>                                                                                                     for
>>                                                                                                     most
>>                                                                                                     uses
>>                                                                                                     cases.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     2.
>>                                                                                                     Session
>>                                                                                                     default
>>                                                                                                     namespace
>>                                                                                                     as
>>                                                                                                     fallback
>>                                                                                                     for
>>                                                                                                     "default-namespace"
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Similar
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     above.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     3.
>>                                                                                                     Late
>>                                                                                                     binding
>>                                                                                                     +
>>                                                                                                     UUID
>>                                                                                                     validation
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     If
>>                                                                                                     I
>>                                                                                                     understand
>>                                                                                                     it
>>                                                                                                     correctly,
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     implementation
>>                                                                                                     already
>>                                                                                                     uses
>>                                                                                                     late
>>                                                                                                     binding.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Generally,
>>                                                                                                     having
>>                                                                                                     UUID
>>                                                                                                     validation
>>                                                                                                     makes
>>                                                                                                     the
>>                                                                                                     setup
>>                                                                                                     more
>>                                                                                                     robust.
>>                                                                                                     Which
>>                                                                                                     is
>>                                                                                                     great.
>>                                                                                                     However,
>>                                                                                                     having
>>                                                                                                     UUID
>>                                                                                                     validation
>>                                                                                                     still
>>                                                                                                     requires
>>                                                                                                     us
>>                                                                                                     to
>>                                                                                                     have
>>                                                                                                     a
>>                                                                                                     portable
>>                                                                                                     table
>>                                                                                                     identifier
>>                                                                                                     specification.
>>                                                                                                     Even
>>                                                                                                     if
>>                                                                                                     we
>>                                                                                                     have
>>                                                                                                     the
>>                                                                                                     UUIDs
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     referenced
>>                                                                                                     tables
>>                                                                                                     from
>>                                                                                                     the
>>                                                                                                     view,
>>                                                                                                     there
>>                                                                                                     simply
>>                                                                                                     isn't
>>                                                                                                     an
>>                                                                                                     interface
>>                                                                                                     that
>>                                                                                                     let's
>>                                                                                                     us
>>                                                                                                     use
>>                                                                                                     those
>>                                                                                                     UUIDs.
>>                                                                                                     The
>>                                                                                                     catalog
>>                                                                                                     interface
>>                                                                                                     is
>>                                                                                                     defined
>>                                                                                                     in
>>                                                                                                     terms
>>                                                                                                     of
>>                                                                                                     table
>>                                                                                                     identifiers.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     So
>>                                                                                                     we
>>                                                                                                     always
>>                                                                                                     require
>>                                                                                                     a
>>                                                                                                     working
>>                                                                                                     catalog
>>                                                                                                     setup
>>                                                                                                     and
>>                                                                                                     suiting
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     to
>>                                                                                                     obtain
>>                                                                                                     the
>>                                                                                                     table
>>                                                                                                     metadata.
>>                                                                                                     We
>>                                                                                                     can
>>                                                                                                     use
>>                                                                                                     the
>>                                                                                                     UUIDs
>>                                                                                                     to
>>                                                                                                     verify
>>                                                                                                     if
>>                                                                                                     we
>>                                                                                                     loaded
>>                                                                                                     the
>>                                                                                                     correct
>>                                                                                                     table.
>>                                                                                                     But
>>                                                                                                     this
>>                                                                                                     can
>>                                                                                                     only
>>                                                                                                     be
>>                                                                                                     done
>>                                                                                                     after
>>                                                                                                     we
>>                                                                                                     used
>>                                                                                                     some
>>                                                                                                     identifier.
>>                                                                                                     Which
>>                                                                                                     means
>>                                                                                                     there
>>                                                                                                     is
>>                                                                                                     no
>>                                                                                                     way
>>                                                                                                     of
>>                                                                                                     using
>>                                                                                                     UUIDs
>>                                                                                                     without
>>                                                                                                     a
>>                                                                                                     functioning
>>                                                                                                     catalog/identifier
>>                                                                                                     setup.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     In
>>                                                                                                     conclusion,
>>                                                                                                     I
>>                                                                                                     prefer
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     behavior
>>                                                                                                     for
>>                                                                                                     "default-catalog"
>>                                                                                                     because
>>                                                                                                     it
>>                                                                                                     is
>>                                                                                                     more
>>                                                                                                     deterministic
>>                                                                                                     in
>>                                                                                                     my
>>                                                                                                     opinion.
>>                                                                                                     And
>>                                                                                                     I
>>                                                                                                     think
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     spec
>>                                                                                                     does
>>                                                                                                     a
>>                                                                                                     good
>>                                                                                                     job
>>                                                                                                     for
>>                                                                                                     multi-engine
>>                                                                                                     table
>>                                                                                                     identifier
>>                                                                                                     resolution.
>>                                                                                                     I
>>                                                                                                     see
>>                                                                                                     the
>>                                                                                                     UUID
>>                                                                                                     validation
>>                                                                                                     more
>>                                                                                                     of
>>                                                                                                     an
>>                                                                                                     additional
>>                                                                                                     hardening
>>                                                                                                     strategy.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Thanks
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Jan
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     On
>>                                                                                                     4/21/25
>>                                                                                                     17:38,
>>                                                                                                     Walaa
>>                                                                                                     Eldin
>>                                                                                                     Moustafa
>>                                                                                                     wrote:
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Thanks
>>                                                                                                     Renjie!
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     The
>>                                                                                                     existing
>>                                                                                                     spec
>>                                                                                                     has
>>                                                                                                     some
>>                                                                                                     guidance
>>                                                                                                     on
>>                                                                                                     resolving
>>                                                                                                     catalogs
>>                                                                                                     on
>>                                                                                                     the
>>                                                                                                     fly
>>                                                                                                     already
>>                                                                                                     (to
>>                                                                                                     address
>>                                                                                                     the
>>                                                                                                     case
>>                                                                                                     of
>>                                                                                                     view
>>                                                                                                     text
>>                                                                                                     with
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     missing
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     part).
>>                                                                                                     The
>>                                                                                                     guidance
>>                                                                                                     is
>>                                                                                                     to
>>                                                                                                     use
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     where
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     stored.
>>                                                                                                     But
>>                                                                                                     I
>>                                                                                                     find
>>                                                                                                     this
>>                                                                                                     rule
>>                                                                                                     hard
>>                                                                                                     to
>>                                                                                                     interpret
>>                                                                                                     or
>>                                                                                                     use.
>>                                                                                                     The
>>                                                                                                     catalog
>>                                                                                                     itself
>>                                                                                                     is
>>                                                                                                     a
>>                                                                                                     logical
>>                                                                                                     construct—such
>>                                                                                                     as
>>                                                                                                     a
>>                                                                                                     federated
>>                                                                                                     catalog
>>                                                                                                     that
>>                                                                                                     delegates
>>                                                                                                     to
>>                                                                                                     multiple
>>                                                                                                     physical
>>                                                                                                     backends
>>                                                                                                     (e.g.,
>>                                                                                                     HMS
>>                                                                                                     and
>>                                                                                                     REST).
>>                                                                                                     In
>>                                                                                                     such
>>                                                                                                     cases,
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     (e.g.,
>>                                                                                                     `my_catalog`
>>                                                                                                     in
>>                                                                                                     `my_catalog.namespace1.table1`)
>>                                                                                                     doesn’t
>>                                                                                                     physically
>>                                                                                                     store
>>                                                                                                     the
>>                                                                                                     tables;
>>                                                                                                     it
>>                                                                                                     only
>>                                                                                                     routes
>>                                                                                                     requests
>>                                                                                                     to
>>                                                                                                     underlying
>>                                                                                                     stores.
>>                                                                                                     Therefore,
>>                                                                                                     defaulting
>>                                                                                                     identifier
>>                                                                                                     resolution
>>                                                                                                     based
>>                                                                                                     on
>>                                                                                                     the
>>                                                                                                     catalog
>>                                                                                                     where
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     is
>>                                                                                                     "stored"
>>                                                                                                     doesn’t
>>                                                                                                     align
>>                                                                                                     with
>>                                                                                                     how
>>                                                                                                     catalogs
>>                                                                                                     actually
>>                                                                                                     behave
>>                                                                                                     in
>>                                                                                                     practice.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     Thanks,
>>                                                                                                     >>>>>>
>>                                                                                                     Walaa.
>>                                                                                                     >>>>>>
>>                                                                                                     >>>>>>
>>                                                                                                     On
>>                                                                                                     Sun,
>>                                                                                                     Apr
>>                                                                                                     20,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     11:17 PM
>>                                                                                                     Renjie
>>                                                                                                     Liu
>>                                                                                                     <li...@gmail.com>
>>                                                                                                     wrote:
>>                                                                                                     >>>>>>>
>>                                                                                                     >>>>>>>
>>                                                                                                     Hi,
>>                                                                                                     Walaa:
>>                                                                                                     >>>>>>>
>>                                                                                                     >>>>>>>
>>                                                                                                     Thanks
>>                                                                                                     for
>>                                                                                                     the
>>                                                                                                     proposal.
>>                                                                                                     >>>>>>>
>>                                                                                                     >>>>>>>
>>                                                                                                     I've
>>                                                                                                     reviewed
>>                                                                                                     the
>>                                                                                                     doc,
>>                                                                                                     but
>>                                                                                                     in
>>                                                                                                     general
>>                                                                                                     I
>>                                                                                                     have
>>                                                                                                     some
>>                                                                                                     concerns
>>                                                                                                     with
>>                                                                                                     resolving
>>                                                                                                     catalog
>>                                                                                                     names
>>                                                                                                     on
>>                                                                                                     the
>>                                                                                                     fly
>>                                                                                                     with
>>                                                                                                     query
>>                                                                                                     engine
>>                                                                                                     defined
>>                                                                                                     catalog
>>                                                                                                     names.
>>                                                                                                     This
>>                                                                                                     introduces
>>                                                                                                     some
>>                                                                                                     flexibility
>>                                                                                                     at
>>                                                                                                     first
>>                                                                                                     glance,
>>                                                                                                     but
>>                                                                                                     also
>>                                                                                                     makes
>>                                                                                                     misconfiguration
>>                                                                                                     difficult
>>                                                                                                     to
>>                                                                                                     explain.
>>                                                                                                     >>>>>>>
>>                                                                                                     >>>>>>>
>>                                                                                                     But
>>                                                                                                     I
>>                                                                                                     agree
>>                                                                                                     with
>>                                                                                                     one
>>                                                                                                     part
>>                                                                                                     that
>>                                                                                                     we
>>                                                                                                     should
>>                                                                                                     store
>>                                                                                                     resolved
>>                                                                                                     table
>>                                                                                                     uuid
>>                                                                                                     in
>>                                                                                                     view
>>                                                                                                     metadata,
>>                                                                                                     as
>>                                                                                                     table/view
>>                                                                                                     renaming
>>                                                                                                     may
>>                                                                                                     introduce
>>                                                                                                     errors
>>                                                                                                     that's
>>                                                                                                     difficult
>>                                                                                                     to
>>                                                                                                     understand
>>                                                                                                     for
>>                                                                                                     user.
>>                                                                                                     >>>>>>>
>>                                                                                                     >>>>>>>
>>                                                                                                     On
>>                                                                                                     Sat,
>>                                                                                                     Apr
>>                                                                                                     19,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     3:02 AM
>>                                                                                                     Walaa
>>                                                                                                     Eldin
>>                                                                                                     Moustafa
>>                                                                                                     <wa...@gmail.com>
>>                                                                                                     wrote:
>>                                                                                                     >>>>>>>>
>>                                                                                                     >>>>>>>>
>>                                                                                                     Hi
>>                                                                                                     Everyone,
>>                                                                                                     >>>>>>>>
>>                                                                                                     >>>>>>>>
>>                                                                                                     Looking
>>                                                                                                     forward
>>                                                                                                     to
>>                                                                                                     keeping
>>                                                                                                     up
>>                                                                                                     the
>>                                                                                                     momentum
>>                                                                                                     and
>>                                                                                                     closing
>>                                                                                                     out
>>                                                                                                     the
>>                                                                                                     MV
>>                                                                                                     spec
>>                                                                                                     as
>>                                                                                                     well.
>>                                                                                                     I’m
>>                                                                                                     hoping
>>                                                                                                     we
>>                                                                                                     can
>>                                                                                                     proceed
>>                                                                                                     to
>>                                                                                                     a
>>                                                                                                     vote
>>                                                                                                     next
>>                                                                                                     week.
>>                                                                                                     >>>>>>>>
>>                                                                                                     >>>>>>>>
>>                                                                                                     Here
>>                                                                                                     is
>>                                                                                                     a
>>                                                                                                     summary
>>                                                                                                     in
>>                                                                                                     case
>>                                                                                                     that
>>                                                                                                     helps.
>>                                                                                                     The
>>                                                                                                     proposal
>>                                                                                                     outlines
>>                                                                                                     a
>>                                                                                                     strategy
>>                                                                                                     for
>>                                                                                                     handling
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     in
>>                                                                                                     Iceberg
>>                                                                                                     view
>>                                                                                                     metadata,
>>                                                                                                     with
>>                                                                                                     the
>>                                                                                                     goal
>>                                                                                                     of
>>                                                                                                     ensuring
>>                                                                                                     correctness,
>>                                                                                                     portability,
>>                                                                                                     and
>>                                                                                                     engine
>>                                                                                                     compatibility.
>>                                                                                                     It
>>                                                                                                     recommends
>>                                                                                                     resolving
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     at
>>                                                                                                     read
>>                                                                                                     time
>>                                                                                                     (late
>>                                                                                                     binding)
>>                                                                                                     rather
>>                                                                                                     than
>>                                                                                                     creation
>>                                                                                                     time,
>>                                                                                                     and
>>                                                                                                     introduces
>>                                                                                                     UUID-based
>>                                                                                                     validation
>>                                                                                                     to
>>                                                                                                     maintain
>>                                                                                                     identity
>>                                                                                                     guarantees
>>                                                                                                     across
>>                                                                                                     engines,
>>                                                                                                     or
>>                                                                                                     sessions.
>>                                                                                                     It
>>                                                                                                     also
>>                                                                                                     revises
>>                                                                                                     how
>>                                                                                                     default-catalog
>>                                                                                                     and
>>                                                                                                     default-namespace
>>                                                                                                     are
>>                                                                                                     handled
>>                                                                                                     (defaulting
>>                                                                                                     both
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     session
>>                                                                                                     context
>>                                                                                                     if
>>                                                                                                     not
>>                                                                                                     explicitly
>>                                                                                                     set)
>>                                                                                                     to
>>                                                                                                     better
>>                                                                                                     align
>>                                                                                                     with
>>                                                                                                     engine
>>                                                                                                     behavior
>>                                                                                                     and
>>                                                                                                     improve
>>                                                                                                     cross-engine
>>                                                                                                     interoperability.
>>                                                                                                     >>>>>>>>
>>                                                                                                     >>>>>>>>
>>                                                                                                     Please
>>                                                                                                     let
>>                                                                                                     me
>>                                                                                                     know
>>                                                                                                     your
>>                                                                                                     thoughts.
>>                                                                                                     >>>>>>>>
>>                                                                                                     >>>>>>>>
>>                                                                                                     Thanks,
>>                                                                                                     >>>>>>>>
>>                                                                                                     Walaa.
>>                                                                                                     >>>>>>>>
>>                                                                                                     >>>>>>>>
>>                                                                                                     >>>>>>>>
>>                                                                                                     >>>>>>>>
>>                                                                                                     On
>>                                                                                                     Wed,
>>                                                                                                     Apr
>>                                                                                                     16,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     2:03 PM
>>                                                                                                     Walaa
>>                                                                                                     Eldin
>>                                                                                                     Moustafa
>>                                                                                                     <wa...@gmail.com>
>>                                                                                                     wrote:
>>                                                                                                     >>>>>>>>>
>>                                                                                                     >>>>>>>>>
>>                                                                                                     Thanks
>>                                                                                                     Eduard
>>                                                                                                     and
>>                                                                                                     Sung!
>>                                                                                                     I
>>                                                                                                     have
>>                                                                                                     addressed
>>                                                                                                     the
>>                                                                                                     comments.
>>                                                                                                     >>>>>>>>>
>>                                                                                                     >>>>>>>>>
>>                                                                                                     One
>>                                                                                                     key
>>                                                                                                     point
>>                                                                                                     to
>>                                                                                                     keep
>>                                                                                                     in
>>                                                                                                     mind
>>                                                                                                     is
>>                                                                                                     that
>>                                                                                                     catalog
>>                                                                                                     names
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     spec
>>                                                                                                     refer
>>                                                                                                     to
>>                                                                                                     logical
>>                                                                                                     catalogs—i.e.,
>>                                                                                                     the
>>                                                                                                     first
>>                                                                                                     part
>>                                                                                                     of
>>                                                                                                     a
>>                                                                                                     three-part
>>                                                                                                     identifier.
>>                                                                                                     These
>>                                                                                                     correspond
>>                                                                                                     to
>>                                                                                                     Spark's
>>                                                                                                     DataSourceV2
>>                                                                                                     catalogs,
>>                                                                                                     Trino
>>                                                                                                     connectors,
>>                                                                                                     and
>>                                                                                                     similar
>>                                                                                                     constructs.
>>                                                                                                     This
>>                                                                                                     is
>>                                                                                                     a
>>                                                                                                     level
>>                                                                                                     of
>>                                                                                                     abstraction
>>                                                                                                     above
>>                                                                                                     physical
>>                                                                                                     catalogs,
>>                                                                                                     which
>>                                                                                                     are
>>                                                                                                     not
>>                                                                                                     referenced
>>                                                                                                     or
>>                                                                                                     used
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     spec.
>>                                                                                                     The
>>                                                                                                     reason
>>                                                                                                     is
>>                                                                                                     that
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     definition/text
>>                                                                                                     itself
>>                                                                                                     refer
>>                                                                                                     to
>>                                                                                                     logical
>>                                                                                                     catalogs,
>>                                                                                                     not
>>                                                                                                     physical
>>                                                                                                     ones
>>                                                                                                     (since
>>                                                                                                     they
>>                                                                                                     interface
>>                                                                                                     directly
>>                                                                                                     with
>>                                                                                                     the
>>                                                                                                     engine
>>                                                                                                     and
>>                                                                                                     not
>>                                                                                                     a
>>                                                                                                     specific
>>                                                                                                     metastore).
>>                                                                                                     >>>>>>>>>
>>                                                                                                     >>>>>>>>>
>>                                                                                                     Thanks,
>>                                                                                                     >>>>>>>>>
>>                                                                                                     Walaa.
>>                                                                                                     >>>>>>>>>
>>                                                                                                     >>>>>>>>>
>>                                                                                                     >>>>>>>>>
>>                                                                                                     On
>>                                                                                                     Wed,
>>                                                                                                     Apr
>>                                                                                                     16,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     6:15 AM
>>                                                                                                     Sung
>>                                                                                                     Yun
>>                                                                                                     <su...@gmail.com>
>>                                                                                                     wrote:
>>                                                                                                     >>>>>>>>>>
>>                                                                                                     >>>>>>>>>>
>>                                                                                                     Thank
>>                                                                                                     you
>>                                                                                                     Walaa
>>                                                                                                     for
>>                                                                                                     the
>>                                                                                                     proposal.
>>                                                                                                     I
>>                                                                                                     think
>>                                                                                                     view
>>                                                                                                     portability
>>                                                                                                     is
>>                                                                                                     a
>>                                                                                                     very
>>                                                                                                     important
>>                                                                                                     topic
>>                                                                                                     for
>>                                                                                                     us
>>                                                                                                     to
>>                                                                                                     continue
>>                                                                                                     discussing
>>                                                                                                     as
>>                                                                                                     it
>>                                                                                                     relies
>>                                                                                                     on
>>                                                                                                     many
>>                                                                                                     assumptions
>>                                                                                                     within
>>                                                                                                     the
>>                                                                                                     data
>>                                                                                                     ecosystem
>>                                                                                                     for
>>                                                                                                     it
>>                                                                                                     to
>>                                                                                                     function
>>                                                                                                     like
>>                                                                                                     you've
>>                                                                                                     highlighted
>>                                                                                                     well
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     document.
>>                                                                                                     >>>>>>>>>>
>>                                                                                                     >>>>>>>>>>
>>                                                                                                     I've
>>                                                                                                     added
>>                                                                                                     a
>>                                                                                                     few
>>                                                                                                     comments
>>                                                                                                     around
>>                                                                                                     how
>>                                                                                                     this
>>                                                                                                     may
>>                                                                                                     impact
>>                                                                                                     the
>>                                                                                                     permission
>>                                                                                                     questions
>>                                                                                                     the
>>                                                                                                     engines
>>                                                                                                     will
>>                                                                                                     be
>>                                                                                                     asking,
>>                                                                                                     and
>>                                                                                                     whether
>>                                                                                                     that
>>                                                                                                     is
>>                                                                                                     the
>>                                                                                                     desired
>>                                                                                                     behavior.
>>                                                                                                     >>>>>>>>>>
>>                                                                                                     >>>>>>>>>>
>>                                                                                                     Sung
>>                                                                                                     >>>>>>>>>>
>>                                                                                                     >>>>>>>>>>
>>                                                                                                     On
>>                                                                                                     Wed,
>>                                                                                                     Apr
>>                                                                                                     16,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     7:32 AM
>>                                                                                                     Eduard
>>                                                                                                     Tudenhöfner
>>                                                                                                     <et...@apache.org>
>>                                                                                                     wrote:
>>                                                                                                     >>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>
>>                                                                                                     Thanks
>>                                                                                                     Walaa
>>                                                                                                     for
>>                                                                                                     tackling
>>                                                                                                     this
>>                                                                                                     problem.
>>                                                                                                     I've
>>                                                                                                     added
>>                                                                                                     a
>>                                                                                                     few
>>                                                                                                     comments
>>                                                                                                     to
>>                                                                                                     get
>>                                                                                                     a
>>                                                                                                     better
>>                                                                                                     understanding
>>                                                                                                     of
>>                                                                                                     how
>>                                                                                                     this
>>                                                                                                     will
>>                                                                                                     look
>>                                                                                                     like
>>                                                                                                     in
>>                                                                                                     the
>>                                                                                                     actual
>>                                                                                                     implementation.
>>                                                                                                     >>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>
>>                                                                                                     Eduard
>>                                                                                                     >>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>
>>                                                                                                     On
>>                                                                                                     Tue,
>>                                                                                                     Apr
>>                                                                                                     15,
>>                                                                                                     2025
>>                                                                                                     at
>>                                                                                                     7:09 PM
>>                                                                                                     Walaa
>>                                                                                                     Eldin
>>                                                                                                     Moustafa
>>                                                                                                     <wa...@gmail.com>
>>                                                                                                     wrote:
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     Hi
>>                                                                                                     Everyone,
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     Starting
>>                                                                                                     this
>>                                                                                                     thread
>>                                                                                                     to
>>                                                                                                     resume
>>                                                                                                     our
>>                                                                                                     discussion
>>                                                                                                     on
>>                                                                                                     how
>>                                                                                                     to
>>                                                                                                     reference
>>                                                                                                     table
>>                                                                                                     identifiers
>>                                                                                                     from
>>                                                                                                     Iceberg
>>                                                                                                     metadata,
>>                                                                                                     a
>>                                                                                                     key
>>                                                                                                     aspect
>>                                                                                                     of
>>                                                                                                     the
>>                                                                                                     view
>>                                                                                                     specification,
>>                                                                                                     particularly
>>                                                                                                     in
>>                                                                                                     relation
>>                                                                                                     to
>>                                                                                                     the
>>                                                                                                     MV
>>                                                                                                     (materialized
>>                                                                                                     view)
>>                                                                                                     extensions.
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     I
>>                                                                                                     had
>>                                                                                                     the
>>                                                                                                     chance
>>                                                                                                     to
>>                                                                                                     speak
>>                                                                                                     offline
>>                                                                                                     with
>>                                                                                                     a
>>                                                                                                     few
>>                                                                                                     community
>>                                                                                                     members
>>                                                                                                     to
>>                                                                                                     better
>>                                                                                                     understand
>>                                                                                                     how
>>                                                                                                     the
>>                                                                                                     current
>>                                                                                                     spec
>>                                                                                                     is
>>                                                                                                     being
>>                                                                                                     interpreted.
>>                                                                                                     Those
>>                                                                                                     conversations
>>                                                                                                     served
>>                                                                                                     as
>>                                                                                                     inputs
>>                                                                                                     to
>>                                                                                                     a
>>                                                                                                     new
>>                                                                                                     proposal
>>                                                                                                     on
>>                                                                                                     how
>>                                                                                                     table
>>                                                                                                     identifier
>>                                                                                                     references
>>                                                                                                     could
>>                                                                                                     be
>>                                                                                                     represented
>>                                                                                                     in
>>                                                                                                     metadata.
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     You
>>                                                                                                     can
>>                                                                                                     find
>>                                                                                                     the
>>                                                                                                     proposal
>>                                                                                                     here
>>                                                                                                     [1].
>>                                                                                                     I
>>                                                                                                     look
>>                                                                                                     forward
>>                                                                                                     to
>>                                                                                                     your
>>                                                                                                     feedback
>>                                                                                                     and
>>                                                                                                     working
>>                                                                                                     together
>>                                                                                                     to
>>                                                                                                     move
>>                                                                                                     this
>>                                                                                                     forward
>>                                                                                                     so
>>                                                                                                     we
>>                                                                                                     can
>>                                                                                                     finalize
>>                                                                                                     the
>>                                                                                                     MV
>>                                                                                                     spec
>>                                                                                                     as
>>                                                                                                     well.
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     [1]
>>                                                                                                     https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     Thanks,
>>                                                                                                     >>>>>>>>>>>>
>>                                                                                                     Walaa.
>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

Let us continue here for the context and continuity of the conversation.

On Thu, May 8, 2025 at 10:55 AM Steven Wu <st...@gmail.com> wrote:

> For the storage table identifier discussion, can we move to Jan's PR on
> materialized view spec?
> https://github.com/apache/iceberg/pull/11041
>
>
> On Thu, May 8, 2025 at 10:40 AM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> It looks like we may not reach alignment on the resolution method in the
>> short term. That said, perhaps we can move forward by aligning on the
>> format used to store table identifiers in metadata instead.
>>
>> The idea is that the same format could work under either resolution
>> model, which would allow us to make progress independently of that debate.
>>
>> Is this storage format of the storage table identifier agreeable to you?
>> * catalog name: optional
>> * namespace: optional
>> * table name: required
>> * UUID: required
>> With the condition that the components included in a table identifier in
>> the metadata should match those used in the view text for that table.
>>
>> The same structure above could apply to storing child tables or view
>> identifiers as part of lineage, if and when that’s needed.
>>
>> For the longer term resolution model, it might make sense to initiate a
>> cross-community discussion.
>>
>> Thanks,
>> Walaa.
>>
>> On Thu, May 8, 2025 at 9:07 AM Steven Wu <st...@gmail.com> wrote:
>>
>>> If a table referenced by the view SQL query is deleted and recreated, is
>>> the regular view still valid? If yes, it won't be correct to use resolved
>>> UUID to validate the name reference. On the materialized view side, we are
>>> aligned that UUID is necessary to detect this scenario and mark the refresh
>>> state "stale" if a referenced table is re-created.
>>>
>>> > esolution using default-catalog and default-namespace does not provide
>>> full determinism,
>>>
>>> With the new proposal, the catalog and namespace depends on the engine's
>>> current context. if the user changes the current context (catalog and
>>> database) via SET/USE , the view SQL query may resolve to a different
>>> table. That is probably a bigger problem.
>>>
>>> With the current view spec, the intention is set in the view creation
>>> time. I would rather have the view producer set the explicit intention
>>> (than the implicit dependency on the engine context at view consumption
>>> phase).
>>>
>>> Also the view SQL can still reference a table with a catalog name
>>> component.  Hence the catalog naming still matters.The new proposal won't
>>> really solve the naming problem.
>>>
>>> On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> Hi Dan,
>>>>
>>>> Thanks for the clarification! I agree that UUIDs should not be stored
>>>> in the SQL definition of the view.
>>>>
>>>> Just to clarify: the proposal isn't about modifying the view definition
>>>> itself, but rather about how table identifiers are represented in Iceberg
>>>> view metadata, which naturally depends on how those identifiers are
>>>> resolved.
>>>>
>>>> For example, in order to store the "storage table identifier" in the
>>>> metadata:
>>>> * Should we apply the same resolution rules based on default-catalog
>>>> and default-namespace?
>>>> * And if that resolution method is considered deterministic enough,
>>>> would you be comfortable not storing the UUID in the storage table
>>>> identifier metadata?
>>>>
>>>> The same questions apply when storing child views or tables as part of
>>>> the view’s lineage metadata. It would be helpful to clarify the resolution
>>>> model and whether UUIDs should be part of that for validation or drift
>>>> detection.
>>>>
>>>> Thanks,
>>>> Walaa.
>>>>
>>>>
>>>>
>>>> On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org> wrote:
>>>>
>>>>> I don't think we want to include the resolved table UUIDs in the view
>>>>> definition, but rather in the storage table state.  You can still resolve
>>>>> whether those drift at some point, but I don't feel like it's a good idea
>>>>> to capture data in the view that we may allow to drift if there isn't any
>>>>> requirement that they match.  This also aligns with the identifier
>>>>> resolution being late binding.
>>>>>
>>>>> -Dan
>>>>>
>>>>> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> Thanks Steven! So would you agree that resolution using
>>>>>> default-catalog and default-namespace does not provide full determinism,
>>>>>> and requires a supporting safety mechanism?
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa.
>>>>>>
>>>>>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> > If the current model is considered deterministic, do you think
>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>
>>>>>>> I am fine with storing UUIDs for table identifiers in the view.
>>>>>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>>>>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>>>>>> fail the query if mismatch.
>>>>>>>
>>>>>>> The UUID change doesn't really change the table identifier
>>>>>>> resolution rule though. It is more of a safety protection.
>>>>>>>
>>>>>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Steven,
>>>>>>>>
>>>>>>>> Thanks for the reply.
>>>>>>>>
>>>>>>>> > I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>> Iceberg view spec.
>>>>>>>>
>>>>>>>> To clarify, I don't believe the proposal is trying to solve catalog
>>>>>>>> naming. What it’s doing is simply this:
>>>>>>>>
>>>>>>>> * Proposing that table names inside views resolve the same way as
>>>>>>>> they do elsewhere (e.g., queries).
>>>>>>>> * Adopting a model that is already widely used and supported in the
>>>>>>>> existing ecosystem, which allows for:
>>>>>>>>     -- Renaming catalog aliases
>>>>>>>>     -- Swapping catalog implementations behind consistent names
>>>>>>>>     -- Having different default catalog names across engines that
>>>>>>>> still point to the same underlying tables
>>>>>>>>
>>>>>>>> These are common patterns in production data lakes. Saying Iceberg
>>>>>>>> views cannot operate in those environments feels unrealistic. In practice,
>>>>>>>> it means the spec breaks down in situations that users encounter regularly.
>>>>>>>>
>>>>>>>> > The recommendation of using engines’ current catalog and database
>>>>>>>> can cause context-dependent resolution results.
>>>>>>>>
>>>>>>>> * As noted in the doc and earlier replies, fixing a catalog name
>>>>>>>> doesn’t actually guarantee determinism either. All the failure scenarios
>>>>>>>> above still apply even when a default-catalog is stored.
>>>>>>>> * The current spec also allows default-catalog to be null, in which
>>>>>>>> case it falls back to the view’s catalog, yet that catalog is determined
>>>>>>>> based on how the view is referenced in the query, which would be considered
>>>>>>>> non-deterministic based on the same criteria you shared.
>>>>>>>> * The only true form of determinism here is UUID-based validation,
>>>>>>>> which protects against silent drift in any resolution model.
>>>>>>>>
>>>>>>>> If the current model is considered deterministic, do you think
>>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>>> Or put another way: Would you be comfortable relying solely on
>>>>>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>>>>>> table, without UUID validation?
>>>>>>>>
>>>>>>>> +1 on involving other communities. I’m happy to help facilitate a
>>>>>>>> cross-community discussion if we aren’t able to reach a resolution here.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Walaa.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>>> Iceberg view spec. I am not convinced that the proposed change will make
>>>>>>>>> the table identifier resolution more clear and portable. The recommendation
>>>>>>>>> of using engines' current catalog and database can cause context dependent
>>>>>>>>> resolution results, which seems non-deterministic to me.
>>>>>>>>>
>>>>>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>>>>>> more inputs from the broader community on this part.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> In Spark, I believe that the USE commands sets the current
>>>>>>>>>> catalog and namespace.  This affects both where the view is created and how
>>>>>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>>>>>> saving the current catalog and namespace into the view
>>>>>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> > I think that's the lesser evil compared to Iceberg specifying
>>>>>>>>>>> how engines should resolve identifiers
>>>>>>>>>>>
>>>>>>>>>>> I think this is also similar to the previous point. It is the
>>>>>>>>>>> other way around. Right now the spec dictates how to resolve (through
>>>>>>>>>>> employing a view-specific `default-catalog` field). The proposal is
>>>>>>>>>>> suggesting to get out of this space and let engines handle it similar
>>>>>>>>>>> to how they handle all identifiers.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>>>
>>>>>>>>>>>> Benny, I think this is a misconception or miscommunication. The
>>>>>>>>>>>> USE command has no impact on the `default-catalog` field. In fact, the
>>>>>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>>>>>> is not the case under the current spec.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> > there is no SQL construct today to explicitly set
>>>>>>>>>>>>> default-catalog
>>>>>>>>>>>>>
>>>>>>>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I generally agree with Dan about requiring consistent catalog
>>>>>>>>>>>>> names.  I think that's the lesser evil compared to Iceberg specifying how
>>>>>>>>>>>>> engines should resolve identifiers.  Another thing to consider is that
>>>>>>>>>>>>> identifier resolution can be very expensive at query validation time if
>>>>>>>>>>>>> identifiers need to be looked up from a bunch of places.  Hopefully, it
>>>>>>>>>>>>> should be possible to define a view in such a way that identifiers can be
>>>>>>>>>>>>> resolved on the first try.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>>>>>> materialized views.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We may be able to converge on this storage format even if we
>>>>>>>>>>>>>> haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>>>>>> identifiers in this way.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm supportive of moving forward with consensus on the
>>>>>>>>>>>>>> identifier storage format. That said, we may continue to run into questions
>>>>>>>>>>>>>> related to resolution during implementation. For example: Should the
>>>>>>>>>>>>>> storage table identifier follow the same default-catalog and
>>>>>>>>>>>>>> default-namespace resolution behavior as other table references?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We can continue having the discussion if the current view
>>>>>>>>>>>>>>> spec needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Rishabh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Correction of typo: both engines seem to set
>>>>>>>>>>>>>>>> default-catalog to the view catalog if it is defined, or to null if the
>>>>>>>>>>>>>>>> view catalog is not defined.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I agree that catalog renaming is an environmental event,
>>>>>>>>>>>>>>>>> but it's a real one that happens frequently in practice.
>>>>>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something
>>>>>>>>>>>>>>>>> as common as catalog renaming feels very restrictive, and could make the
>>>>>>>>>>>>>>>>> spec less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Further, it's worth noting that the table spec is
>>>>>>>>>>>>>>>>> resilient to catalog renaming, but the view spec is not. If we have an
>>>>>>>>>>>>>>>>> opportunity to make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>>>>>> Both specifications are deterministic in their definition,
>>>>>>>>>>>>>>>>> but one is more fragile to environmental changes than the other. Improving
>>>>>>>>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>>>>>>>>>> more portable over time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>>>>>> Today, both engines seem to set default-catalog to null if
>>>>>>>>>>>>>>>>> the view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> > tables inside views remain reachable after a catalog
>>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This problem stems from the exact
>>>>>>>>>>>>>>>>>> environmental/configuration issue that we should not be trying to address.
>>>>>>>>>>>>>>>>>> I don't think we would expect references to survive a catalog rename.
>>>>>>>>>>>>>>>>>> That's not something covered by the spec and needs to be handled separately
>>>>>>>>>>>>>>>>>> as a platform-level migration specific to the affected environment.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The identifier resolution logic is clear and
>>>>>>>>>>>>>>>>>> deterministic.  It should not matter whether an engine resolves and encodes
>>>>>>>>>>>>>>>>>> the default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but rather
>>>>>>>>>>>>>>>>>> view behavior when you start altering the environment around it, which
>>>>>>>>>>>>>>>>>> isn't something we should be trying to define here.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just
>>>>>>>>>>>>>>>>>>> catalog naming consistency. The behavior around default-catalog itself
>>>>>>>>>>>>>>>>>>> introduces resolution inconsistencies even when catalog names are
>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside
>>>>>>>>>>>>>>>>>>> views remain reachable after a catalog rename. But if it is set to a
>>>>>>>>>>>>>>>>>>> non-null value, table references will break.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * default-catalog causes table references inside views
>>>>>>>>>>>>>>>>>>> to be early bound (i.e., bound at view creation time, especially when using
>>>>>>>>>>>>>>>>>>> a non-null value), while table references inside standalone queries are
>>>>>>>>>>>>>>>>>>> late bound (bound at query time). This creates inconsistencies when
>>>>>>>>>>>>>>>>>>> resolving the same table name inside and outside views, even within the
>>>>>>>>>>>>>>>>>>> same job.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the
>>>>>>>>>>>>>>>>>>> spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg
>>>>>>>>>>>>>>>>>>> with that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> How would you suggest approaching the engine-level
>>>>>>>>>>>>>>>>>>> changes required to support the current default-catalog field?
>>>>>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities
>>>>>>>>>>>>>>>>>>> would align around having table resolution behave inconsistently between
>>>>>>>>>>>>>>>>>>> queries and views, or inconsistency between Iceberg and other types of
>>>>>>>>>>>>>>>>>>> views?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I would agree with Jan's summary of why
>>>>>>>>>>>>>>>>>>>> 'default-catalog' was introduced, but I think we need to step back and
>>>>>>>>>>>>>>>>>>>> align on what we are really attempting to support in the spec.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent
>>>>>>>>>>>>>>>>>>>> to address the catalog naming/configuration and would argue that we
>>>>>>>>>>>>>>>>>>>> shouldn't attempt to do that as part of the spec.  Inconsistently named
>>>>>>>>>>>>>>>>>>>> catalogs are a reality, but we should consider that a
>>>>>>>>>>>>>>>>>>>> configuration/environmental issue, not something to solve for in the spec.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We should support and advocate for consistency in
>>>>>>>>>>>>>>>>>>>> catalog naming and define the spec along those lines.  The fact is that
>>>>>>>>>>>>>>>>>>>> with all of the recent work that's gone into making catalogs pluggable, it
>>>>>>>>>>>>>>>>>>>> makes more sense to just register catalog configuration with consistent
>>>>>>>>>>>>>>>>>>>> names (even if you have to duplicate the configuration for supporting
>>>>>>>>>>>>>>>>>>>> existing readers/writers).  I think it's better to provide a path toward
>>>>>>>>>>>>>>>>>>>> consistency than to normalize complicated schemes to workaround the issues
>>>>>>>>>>>>>>>>>>>> caused by environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late
>>>>>>>>>>>>>>>>>>>> binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was
>>>>>>>>>>>>>>>>>>>>> that every query engine uses it to store its session default catalog at the
>>>>>>>>>>>>>>>>>>>>> time of creating the view. This way the view could be reused in another
>>>>>>>>>>>>>>>>>>>>> session. The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Generally we have different environments we want to
>>>>>>>>>>>>>>>>>>>>> support with the view spec:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> When the environment supports it, using consistent
>>>>>>>>>>>>>>>>>>>>> catalog names can have a great benefit for multi-catalog, multi-engine
>>>>>>>>>>>>>>>>>>>>> setups. With consistent catalog names, using the "default-catalog" field
>>>>>>>>>>>>>>>>>>>>> works without any issues.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This can be the case when different query engines
>>>>>>>>>>>>>>>>>>>>> refer to the same physical catalog by different names. This often happens
>>>>>>>>>>>>>>>>>>>>> because different query engines use different strategies to setup the
>>>>>>>>>>>>>>>>>>>>> catalogs. If catalogs have inconsistent naming, using the "default-catalog"
>>>>>>>>>>>>>>>>>>>>> field does not work because it is not guaranteed that the catalog name can
>>>>>>>>>>>>>>>>>>>>> be resolved with another engine. Using the "view catalog" as a fallback is
>>>>>>>>>>>>>>>>>>>>> a better solution for this use case, as it avoids catalog names altogether.
>>>>>>>>>>>>>>>>>>>>> It is however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields
>>>>>>>>>>>>>>>>>>>>> from the view spec completely.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate
>>>>>>>>>>>>>>>>>>>>> view-level default catalogs or namespaces.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly
>>>>>>>>>>>>>>>>>>>>> as they appear in the view SQL text.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation,
>>>>>>>>>>>>>>>>>>>>> it should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the
>>>>>>>>>>>>>>>>>>>>>> confusion the fallback creates. Also just to be clear, at this point and
>>>>>>>>>>>>>>>>>>>>>> after clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I
>>>>>>>>>>>>>>>>>>>>>>> don't think we should design something that says first use this catalog,
>>>>>>>>>>>>>>>>>>>>>>> then that catalog.. or that catalog.  For example, resolving identifiers
>>>>>>>>>>>>>>>>>>>>>>> using default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I
>>>>>>>>>>>>>>>>>>>>>>> see no value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>>>>>> - Interoperability between different engine types is
>>>>>>>>>>>>>>>>>>>>>>> very hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is
>>>>>>>>>>>>>>>>>>>>>>> along the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier
>>>>>>>>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the right approach
>>>>>>>>>>>>>>>>>>>>>>> is really about templating.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers
>>>>>>>>>>>>>>>>>>>>>>> in a view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the
>>>>>>>>>>>>>>>>>>>>>>>> inconsistencies I shared regarding many resolution methods not consistently
>>>>>>>>>>>>>>>>>>>>>>>> being followed in different scenarios (view vs child table resolution,
>>>>>>>>>>>>>>>>>>>>>>>> query vs view resolution)? Note these occur when the default catalog is set
>>>>>>>>>>>>>>>>>>>>>>>> to a non-null value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback
>>>>>>>>>>>>>>>>>>>>>>>>> should be the catalog
>>>>>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really
>>>>>>>>>>>>>>>>>>>>>>>>> matter what the catalog
>>>>>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same
>>>>>>>>>>>>>>>>>>>>>>>>> catalog, this is a
>>>>>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog,
>>>>>>>>>>>>>>>>>>>>>>>>> catalog names
>>>>>>>>>>>>>>>>>>>>>>>>> should be included in the reference name already.
>>>>>>>>>>>>>>>>>>>>>>>>> So no ambiguity
>>>>>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a
>>>>>>>>>>>>>>>>>>>>>>>>> separate problem, which
>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can
>>>>>>>>>>>>>>>>>>>>>>>>> only recommend that
>>>>>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage
>>>>>>>>>>>>>>>>>>>>>>>>> for better
>>>>>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior
>>>>>>>>>>>>>>>>>>>>>>>>> to engine's session
>>>>>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than
>>>>>>>>>>>>>>>>>>>>>>>>> the current fallback
>>>>>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue
>>>>>>>>>>>>>>>>>>>>>>>>> for not conforming
>>>>>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I
>>>>>>>>>>>>>>>>>>>>>>>>> want to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at
>>>>>>>>>>>>>>>>>>>>>>>>> view creation time. It would be circular: the view needs to exist before
>>>>>>>>>>>>>>>>>>>>>>>>> its metadata (and hence default-catalog) can exist. This is visible in
>>>>>>>>>>>>>>>>>>>>>>>>> Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog
>>>>>>>>>>>>>>>>>>>>>>>>> setting would require extending SQL syntax and engine APIs to promote it to
>>>>>>>>>>>>>>>>>>>>>>>>> a first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of
>>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" where the view is going to be created in. It can use
>>>>>>>>>>>>>>>>>>>>>>>>> that catalog to resolve partial table identifiers if "default-catalog" is
>>>>>>>>>>>>>>>>>>>>>>>>> not set.
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where
>>>>>>>>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the
>>>>>>>>>>>>>>>>>>>>>>>>> tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How
>>>>>>>>>>>>>>>>>>>>>>>>> do we validate that a table exists without a default catalog, or do we
>>>>>>>>>>>>>>>>>>>>>>>>> always use the current session catalog?
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding.
>>>>>>>>>>>>>>>>>>>>>>>>> Just to clarify: when I referred to late binding as “similar” to the
>>>>>>>>>>>>>>>>>>>>>>>>> proposal, I was acknowledging the distinction between view-level and
>>>>>>>>>>>>>>>>>>>>>>>>> table-level resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view
>>>>>>>>>>>>>>>>>>>>>>>>> catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to
>>>>>>>>>>>>>>>>>>>>>>>>> distinguish between the physical catalog and it's reference used in SQL
>>>>>>>>>>>>>>>>>>>>>>>>> statements. The important part is that the physical catalog of the view and
>>>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in it's definition stay consistent. You could create
>>>>>>>>>>>>>>>>>>>>>>>>> a view in a given physical catalog by referring to it as "catalogA", as in
>>>>>>>>>>>>>>>>>>>>>>>>> your first point. If you then, given a different setup, refer to the same
>>>>>>>>>>>>>>>>>>>>>>>>> physical catalog as "catalogB" in another session/environment, the behavior
>>>>>>>>>>>>>>>>>>>>>>>>> should still work.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point.
>>>>>>>>>>>>>>>>>>>>>>>>> Late binding applies to the view catalog name and by extension to all
>>>>>>>>>>>>>>>>>>>>>>>>> partial table references when no "default-catalog" is present. Resolving
>>>>>>>>>>>>>>>>>>>>>>>>> the view catalog name at query time is not opposed to storing the view
>>>>>>>>>>>>>>>>>>>>>>>>> metadata in a catalog.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you
>>>>>>>>>>>>>>>>>>>>>>>>> mean.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being
>>>>>>>>>>>>>>>>>>>>>>>>> referenced in a SQL statement. That statement contains the information for
>>>>>>>>>>>>>>>>>>>>>>>>> the query engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT
>>>>>>>>>>>>>>>>>>>>>>>>> * FROM catalogB.namespace.view (after renaming catalogA to catalogB, and
>>>>>>>>>>>>>>>>>>>>>>>>> keeping everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it
>>>>>>>>>>>>>>>>>>>>>>>>> sounds to me like the catalog is resolved at query time, based on how the
>>>>>>>>>>>>>>>>>>>>>>>>> view is referenced, not from any stored metadata. That would imply some
>>>>>>>>>>>>>>>>>>>>>>>>> sort of a late binding behavior (similar to the proposal), as opposed to
>>>>>>>>>>>>>>>>>>>>>>>>> using some catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current
>>>>>>>>>>>>>>>>>>>>>>>>> spec: The view is executed when it's being referenced in a SQL statement.
>>>>>>>>>>>>>>>>>>>>>>>>> That statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog
>>>>>>>>>>>>>>>>>>>>>>>>> can be referenced at execution time. As long as that's the case I would
>>>>>>>>>>>>>>>>>>>>>>>>> assume the view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable
>>>>>>>>>>>>>>>>>>>>>>>>> because the current specification can lead to some unintuitive behavior.
>>>>>>>>>>>>>>>>>>>>>>>>> For example for the following statement:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the
>>>>>>>>>>>>>>>>>>>>>>>>> view spec, because it has the "closure" property. Because of the fact that
>>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" has to be known when executing a view, all the
>>>>>>>>>>>>>>>>>>>>>>>>> information required to resolve the table identifiers is contained in the
>>>>>>>>>>>>>>>>>>>>>>>>> view metadata (and the "view catalog"). I think that if you make the
>>>>>>>>>>>>>>>>>>>>>>>>> identifier resolution dependent on external parameters, it hinders
>>>>>>>>>>>>>>>>>>>>>>>>> portability.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key
>>>>>>>>>>>>>>>>>>>>>>>>> point before going deeper:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you
>>>>>>>>>>>>>>>>>>>>>>>>> help clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view
>>>>>>>>>>>>>>>>>>>>>>>>> catalog” stored, so that an engine knows to fall back to it if
>>>>>>>>>>>>>>>>>>>>>>>>> default-catalog is null?
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we
>>>>>>>>>>>>>>>>>>>>>>>>> can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I
>>>>>>>>>>>>>>>>>>>>>>>>> understood correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the
>>>>>>>>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of "non-determinism".
>>>>>>>>>>>>>>>>>>>>>>>>> You could be running the same query-engine and catalog-setup on different
>>>>>>>>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is rather common), and
>>>>>>>>>>>>>>>>>>>>>>>>> would be getting different results.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view
>>>>>>>>>>>>>>>>>>>>>>>>> always produces the same results. The current behavior has some rough edges
>>>>>>>>>>>>>>>>>>>>>>>>> in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback
>>>>>>>>>>>>>>>>>>>>>>>>> for "default-namespace"
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the
>>>>>>>>>>>>>>>>>>>>>>>>> setup more robust. Which is great. However, having UUID validation still
>>>>>>>>>>>>>>>>>>>>>>>>> requires us to have a portable table identifier specification. Even if we
>>>>>>>>>>>>>>>>>>>>>>>>> have the UUIDs of the referenced tables from the view, there simply isn't
>>>>>>>>>>>>>>>>>>>>>>>>> an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog
>>>>>>>>>>>>>>>>>>>>>>>>> setup and suiting table identifiers to obtain the table metadata. We can
>>>>>>>>>>>>>>>>>>>>>>>>> use the UUIDs to verify if we loaded the correct table. But this can only
>>>>>>>>>>>>>>>>>>>>>>>>> be done after we used some identifier. Which means there is no way of using
>>>>>>>>>>>>>>>>>>>>>>>>> UUIDs without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current
>>>>>>>>>>>>>>>>>>>>>>>>> behavior for "default-catalog" because it is more deterministic in my
>>>>>>>>>>>>>>>>>>>>>>>>> opinion. And I think the current spec does a good job for multi-engine
>>>>>>>>>>>>>>>>>>>>>>>>> table identifier resolution. I see the UUID validation more of an
>>>>>>>>>>>>>>>>>>>>>>>>> additional hardening strategy.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu
>>>>>>>>>>>>>>>>>>>>>>>>> <li...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I
>>>>>>>>>>>>>>>>>>>>>>>>> have some concerns with resolving catalog names on the fly with query
>>>>>>>>>>>>>>>>>>>>>>>>> engine defined catalog names. This introduces some flexibility at first
>>>>>>>>>>>>>>>>>>>>>>>>> glance, but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should
>>>>>>>>>>>>>>>>>>>>>>>>> store resolved table uuid in view metadata, as table/view renaming may
>>>>>>>>>>>>>>>>>>>>>>>>> introduce errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the
>>>>>>>>>>>>>>>>>>>>>>>>> momentum and closing out the MV spec as well. I’m hoping we can proceed to
>>>>>>>>>>>>>>>>>>>>>>>>> a vote next week.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed
>>>>>>>>>>>>>>>>>>>>>>>>> the comments.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that
>>>>>>>>>>>>>>>>>>>>>>>>> catalog names in the spec refer to logical catalogs—i.e., the first part of
>>>>>>>>>>>>>>>>>>>>>>>>> a three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun
>>>>>>>>>>>>>>>>>>>>>>>>> <su...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I
>>>>>>>>>>>>>>>>>>>>>>>>> think view portability is a very important topic for us to continue
>>>>>>>>>>>>>>>>>>>>>>>>> discussing as it relies on many assumptions within the data ecosystem for
>>>>>>>>>>>>>>>>>>>>>>>>> it to function like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how
>>>>>>>>>>>>>>>>>>>>>>>>> this may impact the permission questions the engines will be asking, and
>>>>>>>>>>>>>>>>>>>>>>>>> whether that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this
>>>>>>>>>>>>>>>>>>>>>>>>> problem. I've added a few comments to get a better understanding of how
>>>>>>>>>>>>>>>>>>>>>>>>> this will look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline
>>>>>>>>>>>>>>>>>>>>>>>>> with a few community members to better understand how the current spec is
>>>>>>>>>>>>>>>>>>>>>>>>> being interpreted. Those conversations served as inputs to a new proposal
>>>>>>>>>>>>>>>>>>>>>>>>> on how table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I
>>>>>>>>>>>>>>>>>>>>>>>>> look forward to your feedback and working together to move this forward so
>>>>>>>>>>>>>>>>>>>>>>>>> we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Steven Wu <st...@gmail.com>]

For the storage table identifier discussion, can we move to Jan's PR on
materialized view spec?
https://github.com/apache/iceberg/pull/11041


On Thu, May 8, 2025 at 10:40 AM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> It looks like we may not reach alignment on the resolution method in the
> short term. That said, perhaps we can move forward by aligning on the
> format used to store table identifiers in metadata instead.
>
> The idea is that the same format could work under either resolution model,
> which would allow us to make progress independently of that debate.
>
> Is this storage format of the storage table identifier agreeable to you?
> * catalog name: optional
> * namespace: optional
> * table name: required
> * UUID: required
> With the condition that the components included in a table identifier in
> the metadata should match those used in the view text for that table.
>
> The same structure above could apply to storing child tables or view
> identifiers as part of lineage, if and when that’s needed.
>
> For the longer term resolution model, it might make sense to initiate a
> cross-community discussion.
>
> Thanks,
> Walaa.
>
> On Thu, May 8, 2025 at 9:07 AM Steven Wu <st...@gmail.com> wrote:
>
>> If a table referenced by the view SQL query is deleted and recreated, is
>> the regular view still valid? If yes, it won't be correct to use resolved
>> UUID to validate the name reference. On the materialized view side, we are
>> aligned that UUID is necessary to detect this scenario and mark the refresh
>> state "stale" if a referenced table is re-created.
>>
>> > esolution using default-catalog and default-namespace does not provide
>> full determinism,
>>
>> With the new proposal, the catalog and namespace depends on the engine's
>> current context. if the user changes the current context (catalog and
>> database) via SET/USE , the view SQL query may resolve to a different
>> table. That is probably a bigger problem.
>>
>> With the current view spec, the intention is set in the view creation
>> time. I would rather have the view producer set the explicit intention
>> (than the implicit dependency on the engine context at view consumption
>> phase).
>>
>> Also the view SQL can still reference a table with a catalog name
>> component.  Hence the catalog naming still matters.The new proposal won't
>> really solve the naming problem.
>>
>> On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> Hi Dan,
>>>
>>> Thanks for the clarification! I agree that UUIDs should not be stored in
>>> the SQL definition of the view.
>>>
>>> Just to clarify: the proposal isn't about modifying the view definition
>>> itself, but rather about how table identifiers are represented in Iceberg
>>> view metadata, which naturally depends on how those identifiers are
>>> resolved.
>>>
>>> For example, in order to store the "storage table identifier" in the
>>> metadata:
>>> * Should we apply the same resolution rules based on default-catalog and
>>> default-namespace?
>>> * And if that resolution method is considered deterministic enough,
>>> would you be comfortable not storing the UUID in the storage table
>>> identifier metadata?
>>>
>>> The same questions apply when storing child views or tables as part of
>>> the view’s lineage metadata. It would be helpful to clarify the resolution
>>> model and whether UUIDs should be part of that for validation or drift
>>> detection.
>>>
>>> Thanks,
>>> Walaa.
>>>
>>>
>>>
>>> On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org> wrote:
>>>
>>>> I don't think we want to include the resolved table UUIDs in the view
>>>> definition, but rather in the storage table state.  You can still resolve
>>>> whether those drift at some point, but I don't feel like it's a good idea
>>>> to capture data in the view that we may allow to drift if there isn't any
>>>> requirement that they match.  This also aligns with the identifier
>>>> resolution being late binding.
>>>>
>>>> -Dan
>>>>
>>>> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> Thanks Steven! So would you agree that resolution using
>>>>> default-catalog and default-namespace does not provide full determinism,
>>>>> and requires a supporting safety mechanism?
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> > If the current model is considered deterministic, do you think
>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>
>>>>>> I am fine with storing UUIDs for table identifiers in the view.
>>>>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>>>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>>>>> fail the query if mismatch.
>>>>>>
>>>>>> The UUID change doesn't really change the table identifier resolution
>>>>>> rule though. It is more of a safety protection.
>>>>>>
>>>>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Steven,
>>>>>>>
>>>>>>> Thanks for the reply.
>>>>>>>
>>>>>>> > I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>> Iceberg view spec.
>>>>>>>
>>>>>>> To clarify, I don't believe the proposal is trying to solve catalog
>>>>>>> naming. What it’s doing is simply this:
>>>>>>>
>>>>>>> * Proposing that table names inside views resolve the same way as
>>>>>>> they do elsewhere (e.g., queries).
>>>>>>> * Adopting a model that is already widely used and supported in the
>>>>>>> existing ecosystem, which allows for:
>>>>>>>     -- Renaming catalog aliases
>>>>>>>     -- Swapping catalog implementations behind consistent names
>>>>>>>     -- Having different default catalog names across engines that
>>>>>>> still point to the same underlying tables
>>>>>>>
>>>>>>> These are common patterns in production data lakes. Saying Iceberg
>>>>>>> views cannot operate in those environments feels unrealistic. In practice,
>>>>>>> it means the spec breaks down in situations that users encounter regularly.
>>>>>>>
>>>>>>> > The recommendation of using engines’ current catalog and database
>>>>>>> can cause context-dependent resolution results.
>>>>>>>
>>>>>>> * As noted in the doc and earlier replies, fixing a catalog name
>>>>>>> doesn’t actually guarantee determinism either. All the failure scenarios
>>>>>>> above still apply even when a default-catalog is stored.
>>>>>>> * The current spec also allows default-catalog to be null, in which
>>>>>>> case it falls back to the view’s catalog, yet that catalog is determined
>>>>>>> based on how the view is referenced in the query, which would be considered
>>>>>>> non-deterministic based on the same criteria you shared.
>>>>>>> * The only true form of determinism here is UUID-based validation,
>>>>>>> which protects against silent drift in any resolution model.
>>>>>>>
>>>>>>> If the current model is considered deterministic, do you think
>>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>>> Or put another way: Would you be comfortable relying solely on
>>>>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>>>>> table, without UUID validation?
>>>>>>>
>>>>>>> +1 on involving other communities. I’m happy to help facilitate a
>>>>>>> cross-community discussion if we aren’t able to reach a resolution here.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Walaa.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>>> Iceberg view spec. I am not convinced that the proposed change will make
>>>>>>>> the table identifier resolution more clear and portable. The recommendation
>>>>>>>> of using engines' current catalog and database can cause context dependent
>>>>>>>> resolution results, which seems non-deterministic to me.
>>>>>>>>
>>>>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>>>>> more inputs from the broader community on this part.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> In Spark, I believe that the USE commands sets the current catalog
>>>>>>>>> and namespace.  This affects both where the view is created and how
>>>>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>>>>> saving the current catalog and namespace into the view
>>>>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>>>>
>>>>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> > I think that's the lesser evil compared to Iceberg specifying
>>>>>>>>>> how engines should resolve identifiers
>>>>>>>>>>
>>>>>>>>>> I think this is also similar to the previous point. It is the
>>>>>>>>>> other way around. Right now the spec dictates how to resolve (through
>>>>>>>>>> employing a view-specific `default-catalog` field). The proposal is
>>>>>>>>>> suggesting to get out of this space and let engines handle it similar
>>>>>>>>>> to how they handle all identifiers.
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>>
>>>>>>>>>>> Benny, I think this is a misconception or miscommunication. The
>>>>>>>>>>> USE command has no impact on the `default-catalog` field. In fact, the
>>>>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>>>>> is not the case under the current spec.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> > there is no SQL construct today to explicitly set
>>>>>>>>>>>> default-catalog
>>>>>>>>>>>>
>>>>>>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>>>
>>>>>>>>>>>> I generally agree with Dan about requiring consistent catalog
>>>>>>>>>>>> names.  I think that's the lesser evil compared to Iceberg specifying how
>>>>>>>>>>>> engines should resolve identifiers.  Another thing to consider is that
>>>>>>>>>>>> identifier resolution can be very expensive at query validation time if
>>>>>>>>>>>> identifiers need to be looked up from a bunch of places.  Hopefully, it
>>>>>>>>>>>> should be possible to define a view in such a way that identifiers can be
>>>>>>>>>>>> resolved on the first try.
>>>>>>>>>>>>
>>>>>>>>>>>> Benny
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>>>>
>>>>>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>>>>> materialized views.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We may be able to converge on this storage format even if we
>>>>>>>>>>>>> haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>>>>> identifiers in this way.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm supportive of moving forward with consensus on the
>>>>>>>>>>>>> identifier storage format. That said, we may continue to run into questions
>>>>>>>>>>>>> related to resolution during implementation. For example: Should the
>>>>>>>>>>>>> storage table identifier follow the same default-catalog and
>>>>>>>>>>>>> default-namespace resolution behavior as other table references?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We can continue having the discussion if the current view
>>>>>>>>>>>>>> spec needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Rishabh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Correction of typo: both engines seem to set default-catalog
>>>>>>>>>>>>>>> to the view catalog if it is defined, or to null if the view catalog is not
>>>>>>>>>>>>>>> defined.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I agree that catalog renaming is an environmental event,
>>>>>>>>>>>>>>>> but it's a real one that happens frequently in practice.
>>>>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something
>>>>>>>>>>>>>>>> as common as catalog renaming feels very restrictive, and could make the
>>>>>>>>>>>>>>>> spec less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Further, it's worth noting that the table spec is resilient
>>>>>>>>>>>>>>>> to catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>>>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>>>>> Both specifications are deterministic in their definition,
>>>>>>>>>>>>>>>> but one is more fragile to environmental changes than the other. Improving
>>>>>>>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>>>>>>>>> more portable over time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>>>>> Today, both engines seem to set default-catalog to null if
>>>>>>>>>>>>>>>> the view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> > tables inside views remain reachable after a catalog
>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This problem stems from the exact
>>>>>>>>>>>>>>>>> environmental/configuration issue that we should not be trying to address.
>>>>>>>>>>>>>>>>> I don't think we would expect references to survive a catalog rename.
>>>>>>>>>>>>>>>>> That's not something covered by the spec and needs to be handled separately
>>>>>>>>>>>>>>>>> as a platform-level migration specific to the affected environment.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The identifier resolution logic is clear and
>>>>>>>>>>>>>>>>> deterministic.  It should not matter whether an engine resolves and encodes
>>>>>>>>>>>>>>>>> the default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but rather
>>>>>>>>>>>>>>>>> view behavior when you start altering the environment around it, which
>>>>>>>>>>>>>>>>> isn't something we should be trying to define here.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just
>>>>>>>>>>>>>>>>>> catalog naming consistency. The behavior around default-catalog itself
>>>>>>>>>>>>>>>>>> introduces resolution inconsistencies even when catalog names are
>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside
>>>>>>>>>>>>>>>>>> views remain reachable after a catalog rename. But if it is set to a
>>>>>>>>>>>>>>>>>> non-null value, table references will break.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * default-catalog causes table references inside views to
>>>>>>>>>>>>>>>>>> be early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the
>>>>>>>>>>>>>>>>>> spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg
>>>>>>>>>>>>>>>>>> with that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> How would you suggest approaching the engine-level
>>>>>>>>>>>>>>>>>> changes required to support the current default-catalog field?
>>>>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities
>>>>>>>>>>>>>>>>>> would align around having table resolution behave inconsistently between
>>>>>>>>>>>>>>>>>> queries and views, or inconsistency between Iceberg and other types of
>>>>>>>>>>>>>>>>>> views?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I would agree with Jan's summary of why
>>>>>>>>>>>>>>>>>>> 'default-catalog' was introduced, but I think we need to step back and
>>>>>>>>>>>>>>>>>>> align on what we are really attempting to support in the spec.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent to
>>>>>>>>>>>>>>>>>>> address the catalog naming/configuration and would argue that we shouldn't
>>>>>>>>>>>>>>>>>>> attempt to do that as part of the spec.  Inconsistently named catalogs are
>>>>>>>>>>>>>>>>>>> a reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We should support and advocate for consistency in
>>>>>>>>>>>>>>>>>>> catalog naming and define the spec along those lines.  The fact is that
>>>>>>>>>>>>>>>>>>> with all of the recent work that's gone into making catalogs pluggable, it
>>>>>>>>>>>>>>>>>>> makes more sense to just register catalog configuration with consistent
>>>>>>>>>>>>>>>>>>> names (even if you have to duplicate the configuration for supporting
>>>>>>>>>>>>>>>>>>> existing readers/writers).  I think it's better to provide a path toward
>>>>>>>>>>>>>>>>>>> consistency than to normalize complicated schemes to workaround the issues
>>>>>>>>>>>>>>>>>>> caused by environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late
>>>>>>>>>>>>>>>>>>> binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was
>>>>>>>>>>>>>>>>>>>> that every query engine uses it to store its session default catalog at the
>>>>>>>>>>>>>>>>>>>> time of creating the view. This way the view could be reused in another
>>>>>>>>>>>>>>>>>>>> session. The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Generally we have different environments we want to
>>>>>>>>>>>>>>>>>>>> support with the view spec:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> When the environment supports it, using consistent
>>>>>>>>>>>>>>>>>>>> catalog names can have a great benefit for multi-catalog, multi-engine
>>>>>>>>>>>>>>>>>>>> setups. With consistent catalog names, using the "default-catalog" field
>>>>>>>>>>>>>>>>>>>> works without any issues.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This can be the case when different query engines refer
>>>>>>>>>>>>>>>>>>>> to the same physical catalog by different names. This often happens because
>>>>>>>>>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields
>>>>>>>>>>>>>>>>>>>> from the view spec completely.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate
>>>>>>>>>>>>>>>>>>>> view-level default catalogs or namespaces.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly
>>>>>>>>>>>>>>>>>>>> as they appear in the view SQL text.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation,
>>>>>>>>>>>>>>>>>>>> it should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion
>>>>>>>>>>>>>>>>>>>>> the fallback creates. Also just to be clear, at this point and after
>>>>>>>>>>>>>>>>>>>>> clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I
>>>>>>>>>>>>>>>>>>>>>> don't think we should design something that says first use this catalog,
>>>>>>>>>>>>>>>>>>>>>> then that catalog.. or that catalog.  For example, resolving identifiers
>>>>>>>>>>>>>>>>>>>>>> using default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see
>>>>>>>>>>>>>>>>>>>>>> no value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>>>>> - Interoperability between different engine types is
>>>>>>>>>>>>>>>>>>>>>> very hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is
>>>>>>>>>>>>>>>>>>>>>> along the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier
>>>>>>>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the right approach
>>>>>>>>>>>>>>>>>>>>>> is really about templating.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in
>>>>>>>>>>>>>>>>>>>>>> a view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies
>>>>>>>>>>>>>>>>>>>>>>> I shared regarding many resolution methods not consistently being followed
>>>>>>>>>>>>>>>>>>>>>>> in different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback
>>>>>>>>>>>>>>>>>>>>>>>> should be the catalog
>>>>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter
>>>>>>>>>>>>>>>>>>>>>>>> what the catalog
>>>>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same
>>>>>>>>>>>>>>>>>>>>>>>> catalog, this is a
>>>>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog,
>>>>>>>>>>>>>>>>>>>>>>>> catalog names
>>>>>>>>>>>>>>>>>>>>>>>> should be included in the reference name already.
>>>>>>>>>>>>>>>>>>>>>>>> So no ambiguity
>>>>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a
>>>>>>>>>>>>>>>>>>>>>>>> separate problem, which
>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can
>>>>>>>>>>>>>>>>>>>>>>>> only recommend that
>>>>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage
>>>>>>>>>>>>>>>>>>>>>>>> for better
>>>>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than
>>>>>>>>>>>>>>>>>>>>>>>> the current fallback
>>>>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue
>>>>>>>>>>>>>>>>>>>>>>>> for not conforming
>>>>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I
>>>>>>>>>>>>>>>>>>>>>>>> want to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at
>>>>>>>>>>>>>>>>>>>>>>>> view creation time. It would be circular: the view needs to exist before
>>>>>>>>>>>>>>>>>>>>>>>> its metadata (and hence default-catalog) can exist. This is visible in
>>>>>>>>>>>>>>>>>>>>>>>> Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog
>>>>>>>>>>>>>>>>>>>>>>>> setting would require extending SQL syntax and engine APIs to promote it to
>>>>>>>>>>>>>>>>>>>>>>>> a first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the
>>>>>>>>>>>>>>>>>>>>>>>> "view catalog" where the view is going to be created in. It can use that
>>>>>>>>>>>>>>>>>>>>>>>> catalog to resolve partial table identifiers if "default-catalog" is not
>>>>>>>>>>>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where
>>>>>>>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the
>>>>>>>>>>>>>>>>>>>>>>>> tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How
>>>>>>>>>>>>>>>>>>>>>>>> do we validate that a table exists without a default catalog, or do we
>>>>>>>>>>>>>>>>>>>>>>>> always use the current session catalog?
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding.
>>>>>>>>>>>>>>>>>>>>>>>> Just to clarify: when I referred to late binding as “similar” to the
>>>>>>>>>>>>>>>>>>>>>>>> proposal, I was acknowledging the distinction between view-level and
>>>>>>>>>>>>>>>>>>>>>>>> table-level resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view
>>>>>>>>>>>>>>>>>>>>>>>> catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to
>>>>>>>>>>>>>>>>>>>>>>>> distinguish between the physical catalog and it's reference used in SQL
>>>>>>>>>>>>>>>>>>>>>>>> statements. The important part is that the physical catalog of the view and
>>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in it's definition stay consistent. You could create
>>>>>>>>>>>>>>>>>>>>>>>> a view in a given physical catalog by referring to it as "catalogA", as in
>>>>>>>>>>>>>>>>>>>>>>>> your first point. If you then, given a different setup, refer to the same
>>>>>>>>>>>>>>>>>>>>>>>> physical catalog as "catalogB" in another session/environment, the behavior
>>>>>>>>>>>>>>>>>>>>>>>> should still work.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late
>>>>>>>>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension to all partial
>>>>>>>>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. Resolving the view
>>>>>>>>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the view metadata in a
>>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you
>>>>>>>>>>>>>>>>>>>>>>>> mean.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being
>>>>>>>>>>>>>>>>>>>>>>>> referenced in a SQL statement. That statement contains the information for
>>>>>>>>>>>>>>>>>>>>>>>> the query engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT
>>>>>>>>>>>>>>>>>>>>>>>> * FROM catalogB.namespace.view (after renaming catalogA to catalogB, and
>>>>>>>>>>>>>>>>>>>>>>>> keeping everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it
>>>>>>>>>>>>>>>>>>>>>>>> sounds to me like the catalog is resolved at query time, based on how the
>>>>>>>>>>>>>>>>>>>>>>>> view is referenced, not from any stored metadata. That would imply some
>>>>>>>>>>>>>>>>>>>>>>>> sort of a late binding behavior (similar to the proposal), as opposed to
>>>>>>>>>>>>>>>>>>>>>>>> using some catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current
>>>>>>>>>>>>>>>>>>>>>>>> spec: The view is executed when it's being referenced in a SQL statement.
>>>>>>>>>>>>>>>>>>>>>>>> That statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can
>>>>>>>>>>>>>>>>>>>>>>>> be referenced at execution time. As long as that's the case I would assume
>>>>>>>>>>>>>>>>>>>>>>>> the view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because
>>>>>>>>>>>>>>>>>>>>>>>> the current specification can lead to some unintuitive behavior. For
>>>>>>>>>>>>>>>>>>>>>>>> example for the following statement:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the
>>>>>>>>>>>>>>>>>>>>>>>> view spec, because it has the "closure" property. Because of the fact that
>>>>>>>>>>>>>>>>>>>>>>>> the "view catalog" has to be known when executing a view, all the
>>>>>>>>>>>>>>>>>>>>>>>> information required to resolve the table identifiers is contained in the
>>>>>>>>>>>>>>>>>>>>>>>> view metadata (and the "view catalog"). I think that if you make the
>>>>>>>>>>>>>>>>>>>>>>>> identifier resolution dependent on external parameters, it hinders
>>>>>>>>>>>>>>>>>>>>>>>> portability.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point
>>>>>>>>>>>>>>>>>>>>>>>> before going deeper:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you
>>>>>>>>>>>>>>>>>>>>>>>> help clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog”
>>>>>>>>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if default-catalog is
>>>>>>>>>>>>>>>>>>>>>>>> null?
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we
>>>>>>>>>>>>>>>>>>>>>>>> can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the
>>>>>>>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of "non-determinism".
>>>>>>>>>>>>>>>>>>>>>>>> You could be running the same query-engine and catalog-setup on different
>>>>>>>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is rather common), and
>>>>>>>>>>>>>>>>>>>>>>>> would be getting different results.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view
>>>>>>>>>>>>>>>>>>>>>>>> always produces the same results. The current behavior has some rough edges
>>>>>>>>>>>>>>>>>>>>>>>> in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the
>>>>>>>>>>>>>>>>>>>>>>>> setup more robust. Which is great. However, having UUID validation still
>>>>>>>>>>>>>>>>>>>>>>>> requires us to have a portable table identifier specification. Even if we
>>>>>>>>>>>>>>>>>>>>>>>> have the UUIDs of the referenced tables from the view, there simply isn't
>>>>>>>>>>>>>>>>>>>>>>>> an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup
>>>>>>>>>>>>>>>>>>>>>>>> and suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior
>>>>>>>>>>>>>>>>>>>>>>>> for "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I
>>>>>>>>>>>>>>>>>>>>>>>> have some concerns with resolving catalog names on the fly with query
>>>>>>>>>>>>>>>>>>>>>>>> engine defined catalog names. This introduces some flexibility at first
>>>>>>>>>>>>>>>>>>>>>>>> glance, but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should
>>>>>>>>>>>>>>>>>>>>>>>> store resolved table uuid in view metadata, as table/view renaming may
>>>>>>>>>>>>>>>>>>>>>>>> introduce errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum
>>>>>>>>>>>>>>>>>>>>>>>> and closing out the MV spec as well. I’m hoping we can proceed to a vote
>>>>>>>>>>>>>>>>>>>>>>>> next week.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed
>>>>>>>>>>>>>>>>>>>>>>>> the comments.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that
>>>>>>>>>>>>>>>>>>>>>>>> catalog names in the spec refer to logical catalogs—i.e., the first part of
>>>>>>>>>>>>>>>>>>>>>>>> a three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I
>>>>>>>>>>>>>>>>>>>>>>>> think view portability is a very important topic for us to continue
>>>>>>>>>>>>>>>>>>>>>>>> discussing as it relies on many assumptions within the data ecosystem for
>>>>>>>>>>>>>>>>>>>>>>>> it to function like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how
>>>>>>>>>>>>>>>>>>>>>>>> this may impact the permission questions the engines will be asking, and
>>>>>>>>>>>>>>>>>>>>>>>> whether that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem.
>>>>>>>>>>>>>>>>>>>>>>>> I've added a few comments to get a better understanding of how this will
>>>>>>>>>>>>>>>>>>>>>>>> look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with
>>>>>>>>>>>>>>>>>>>>>>>> a few community members to better understand how the current spec is being
>>>>>>>>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I
>>>>>>>>>>>>>>>>>>>>>>>> look forward to your feedback and working together to move this forward so
>>>>>>>>>>>>>>>>>>>>>>>> we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

It looks like we may not reach alignment on the resolution method in the
short term. That said, perhaps we can move forward by aligning on the
format used to store table identifiers in metadata instead.

The idea is that the same format could work under either resolution model,
which would allow us to make progress independently of that debate.

Is this storage format of the storage table identifier agreeable to you?
* catalog name: optional
* namespace: optional
* table name: required
* UUID: required
With the condition that the components included in a table identifier in
the metadata should match those used in the view text for that table.

The same structure above could apply to storing child tables or view
identifiers as part of lineage, if and when that’s needed.

For the longer term resolution model, it might make sense to initiate a
cross-community discussion.

Thanks,
Walaa.

On Thu, May 8, 2025 at 9:07 AM Steven Wu <st...@gmail.com> wrote:

> If a table referenced by the view SQL query is deleted and recreated, is
> the regular view still valid? If yes, it won't be correct to use resolved
> UUID to validate the name reference. On the materialized view side, we are
> aligned that UUID is necessary to detect this scenario and mark the refresh
> state "stale" if a referenced table is re-created.
>
> > esolution using default-catalog and default-namespace does not provide
> full determinism,
>
> With the new proposal, the catalog and namespace depends on the engine's
> current context. if the user changes the current context (catalog and
> database) via SET/USE , the view SQL query may resolve to a different
> table. That is probably a bigger problem.
>
> With the current view spec, the intention is set in the view creation
> time. I would rather have the view producer set the explicit intention
> (than the implicit dependency on the engine context at view consumption
> phase).
>
> Also the view SQL can still reference a table with a catalog name
> component.  Hence the catalog naming still matters.The new proposal won't
> really solve the naming problem.
>
> On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa <wa...@gmail.com>
> wrote:
>
>> Hi Dan,
>>
>> Thanks for the clarification! I agree that UUIDs should not be stored in
>> the SQL definition of the view.
>>
>> Just to clarify: the proposal isn't about modifying the view definition
>> itself, but rather about how table identifiers are represented in Iceberg
>> view metadata, which naturally depends on how those identifiers are
>> resolved.
>>
>> For example, in order to store the "storage table identifier" in the
>> metadata:
>> * Should we apply the same resolution rules based on default-catalog and
>> default-namespace?
>> * And if that resolution method is considered deterministic enough, would
>> you be comfortable not storing the UUID in the storage table identifier
>> metadata?
>>
>> The same questions apply when storing child views or tables as part of
>> the view’s lineage metadata. It would be helpful to clarify the resolution
>> model and whether UUIDs should be part of that for validation or drift
>> detection.
>>
>> Thanks,
>> Walaa.
>>
>>
>>
>> On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org> wrote:
>>
>>> I don't think we want to include the resolved table UUIDs in the view
>>> definition, but rather in the storage table state.  You can still resolve
>>> whether those drift at some point, but I don't feel like it's a good idea
>>> to capture data in the view that we may allow to drift if there isn't any
>>> requirement that they match.  This also aligns with the identifier
>>> resolution being late binding.
>>>
>>> -Dan
>>>
>>> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> Thanks Steven! So would you agree that resolution using default-catalog
>>>> and default-namespace does not provide full determinism, and requires a
>>>> supporting safety mechanism?
>>>>
>>>> Thanks,
>>>> Walaa.
>>>>
>>>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com> wrote:
>>>>
>>>>> > If the current model is considered deterministic, do you think
>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>
>>>>> I am fine with storing UUIDs for table identifiers in the view.
>>>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>>>> fail the query if mismatch.
>>>>>
>>>>> The UUID change doesn't really change the table identifier resolution
>>>>> rule though. It is more of a safety protection.
>>>>>
>>>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> Hi Steven,
>>>>>>
>>>>>> Thanks for the reply.
>>>>>>
>>>>>> > I agree with Dan that we shouldn't solve catalog naming in the
>>>>>> Iceberg view spec.
>>>>>>
>>>>>> To clarify, I don't believe the proposal is trying to solve catalog
>>>>>> naming. What it’s doing is simply this:
>>>>>>
>>>>>> * Proposing that table names inside views resolve the same way as
>>>>>> they do elsewhere (e.g., queries).
>>>>>> * Adopting a model that is already widely used and supported in the
>>>>>> existing ecosystem, which allows for:
>>>>>>     -- Renaming catalog aliases
>>>>>>     -- Swapping catalog implementations behind consistent names
>>>>>>     -- Having different default catalog names across engines that
>>>>>> still point to the same underlying tables
>>>>>>
>>>>>> These are common patterns in production data lakes. Saying Iceberg
>>>>>> views cannot operate in those environments feels unrealistic. In practice,
>>>>>> it means the spec breaks down in situations that users encounter regularly.
>>>>>>
>>>>>> > The recommendation of using engines’ current catalog and database
>>>>>> can cause context-dependent resolution results.
>>>>>>
>>>>>> * As noted in the doc and earlier replies, fixing a catalog name
>>>>>> doesn’t actually guarantee determinism either. All the failure scenarios
>>>>>> above still apply even when a default-catalog is stored.
>>>>>> * The current spec also allows default-catalog to be null, in which
>>>>>> case it falls back to the view’s catalog, yet that catalog is determined
>>>>>> based on how the view is referenced in the query, which would be considered
>>>>>> non-deterministic based on the same criteria you shared.
>>>>>> * The only true form of determinism here is UUID-based validation,
>>>>>> which protects against silent drift in any resolution model.
>>>>>>
>>>>>> If the current model is considered deterministic, do you think
>>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>>> Or put another way: Would you be comfortable relying solely on
>>>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>>>> table, without UUID validation?
>>>>>>
>>>>>> +1 on involving other communities. I’m happy to help facilitate a
>>>>>> cross-community discussion if we aren’t able to reach a resolution here.
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I agree with Dan that we shouldn't solve catalog naming in the
>>>>>>> Iceberg view spec. I am not convinced that the proposed change will make
>>>>>>> the table identifier resolution more clear and portable. The recommendation
>>>>>>> of using engines' current catalog and database can cause context dependent
>>>>>>> resolution results, which seems non-deterministic to me.
>>>>>>>
>>>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>>>> more inputs from the broader community on this part.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:
>>>>>>>
>>>>>>>> In Spark, I believe that the USE commands sets the current catalog
>>>>>>>> and namespace.  This affects both where the view is created and how
>>>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>>>> saving the current catalog and namespace into the view
>>>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>>>
>>>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> > I think that's the lesser evil compared to Iceberg specifying
>>>>>>>>> how engines should resolve identifiers
>>>>>>>>>
>>>>>>>>> I think this is also similar to the previous point. It is the
>>>>>>>>> other way around. Right now the spec dictates how to resolve (through
>>>>>>>>> employing a view-specific `default-catalog` field). The proposal is
>>>>>>>>> suggesting to get out of this space and let engines handle it similar
>>>>>>>>> to how they handle all identifiers.
>>>>>>>>>
>>>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>
>>>>>>>>>> Benny, I think this is a misconception or miscommunication. The
>>>>>>>>>> USE command has no impact on the `default-catalog` field. In fact, the
>>>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>>>> is not the case under the current spec.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> > there is no SQL construct today to explicitly set
>>>>>>>>>>> default-catalog
>>>>>>>>>>>
>>>>>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>>
>>>>>>>>>>> I generally agree with Dan about requiring consistent catalog
>>>>>>>>>>> names.  I think that's the lesser evil compared to Iceberg specifying how
>>>>>>>>>>> engines should resolve identifiers.  Another thing to consider is that
>>>>>>>>>>> identifier resolution can be very expensive at query validation time if
>>>>>>>>>>> identifiers need to be looked up from a bunch of places.  Hopefully, it
>>>>>>>>>>> should be possible to define a view in such a way that identifiers can be
>>>>>>>>>>> resolved on the first try.
>>>>>>>>>>>
>>>>>>>>>>> Benny
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>>>
>>>>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>>>> materialized views.
>>>>>>>>>>>>
>>>>>>>>>>>> We may be able to converge on this storage format even if we
>>>>>>>>>>>> haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>>>> identifiers in this way.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm supportive of moving forward with consensus on the
>>>>>>>>>>>> identifier storage format. That said, we may continue to run into questions
>>>>>>>>>>>> related to resolution during implementation. For example: Should the
>>>>>>>>>>>> storage table identifier follow the same default-catalog and
>>>>>>>>>>>> default-namespace resolution behavior as other table references?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We can continue having the discussion if the current view spec
>>>>>>>>>>>>> needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Rishabh
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Correction of typo: both engines seem to set default-catalog
>>>>>>>>>>>>>> to the view catalog if it is defined, or to null if the view catalog is not
>>>>>>>>>>>>>> defined.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I agree that catalog renaming is an environmental event, but
>>>>>>>>>>>>>>> it's a real one that happens frequently in practice.
>>>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something as
>>>>>>>>>>>>>>> common as catalog renaming feels very restrictive, and could make the spec
>>>>>>>>>>>>>>> less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Further, it's worth noting that the table spec is resilient
>>>>>>>>>>>>>>> to catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>>>> Both specifications are deterministic in their definition,
>>>>>>>>>>>>>>> but one is more fragile to environmental changes than the other. Improving
>>>>>>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>>>>>>>> more portable over time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>>>> Today, both engines seem to set default-catalog to null if
>>>>>>>>>>>>>>> the view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> > tables inside views remain reachable after a catalog
>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This problem stems from the exact
>>>>>>>>>>>>>>>> environmental/configuration issue that we should not be trying to address.
>>>>>>>>>>>>>>>> I don't think we would expect references to survive a catalog rename.
>>>>>>>>>>>>>>>> That's not something covered by the spec and needs to be handled separately
>>>>>>>>>>>>>>>> as a platform-level migration specific to the affected environment.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The identifier resolution logic is clear and
>>>>>>>>>>>>>>>> deterministic.  It should not matter whether an engine resolves and encodes
>>>>>>>>>>>>>>>> the default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but rather
>>>>>>>>>>>>>>>> view behavior when you start altering the environment around it, which
>>>>>>>>>>>>>>>> isn't something we should be trying to define here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just
>>>>>>>>>>>>>>>>> catalog naming consistency. The behavior around default-catalog itself
>>>>>>>>>>>>>>>>> introduces resolution inconsistencies even when catalog names are
>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside views
>>>>>>>>>>>>>>>>> remain reachable after a catalog rename. But if it is set to a non-null
>>>>>>>>>>>>>>>>> value, table references will break.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * default-catalog causes table references inside views to
>>>>>>>>>>>>>>>>> be early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the
>>>>>>>>>>>>>>>>> spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg
>>>>>>>>>>>>>>>>> with that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities would
>>>>>>>>>>>>>>>>> align around having table resolution behave inconsistently between queries
>>>>>>>>>>>>>>>>> and views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog'
>>>>>>>>>>>>>>>>>> was introduced, but I think we need to step back and align on what we are
>>>>>>>>>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent to
>>>>>>>>>>>>>>>>>> address the catalog naming/configuration and would argue that we shouldn't
>>>>>>>>>>>>>>>>>> attempt to do that as part of the spec.  Inconsistently named catalogs are
>>>>>>>>>>>>>>>>>> a reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We should support and advocate for consistency in catalog
>>>>>>>>>>>>>>>>>> naming and define the spec along those lines.  The fact is that with all of
>>>>>>>>>>>>>>>>>> the recent work that's gone into making catalogs pluggable, it makes more
>>>>>>>>>>>>>>>>>> sense to just register catalog configuration with consistent names (even if
>>>>>>>>>>>>>>>>>> you have to duplicate the configuration for supporting existing
>>>>>>>>>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late
>>>>>>>>>>>>>>>>>> binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was
>>>>>>>>>>>>>>>>>>> that every query engine uses it to store its session default catalog at the
>>>>>>>>>>>>>>>>>>> time of creating the view. This way the view could be reused in another
>>>>>>>>>>>>>>>>>>> session. The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Generally we have different environments we want to
>>>>>>>>>>>>>>>>>>> support with the view spec:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> When the environment supports it, using consistent
>>>>>>>>>>>>>>>>>>> catalog names can have a great benefit for multi-catalog, multi-engine
>>>>>>>>>>>>>>>>>>> setups. With consistent catalog names, using the "default-catalog" field
>>>>>>>>>>>>>>>>>>> works without any issues.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This can be the case when different query engines refer
>>>>>>>>>>>>>>>>>>> to the same physical catalog by different names. This often happens because
>>>>>>>>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields
>>>>>>>>>>>>>>>>>>> from the view spec completely.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level
>>>>>>>>>>>>>>>>>>> default catalogs or namespaces.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as
>>>>>>>>>>>>>>>>>>> they appear in the view SQL text.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation,
>>>>>>>>>>>>>>>>>>> it should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion
>>>>>>>>>>>>>>>>>>>> the fallback creates. Also just to be clear, at this point and after
>>>>>>>>>>>>>>>>>>>> clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't
>>>>>>>>>>>>>>>>>>>>> think we should design something that says first use this catalog, then
>>>>>>>>>>>>>>>>>>>>> that catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see
>>>>>>>>>>>>>>>>>>>>> no value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>>>> - Interoperability between different engine types is
>>>>>>>>>>>>>>>>>>>>> very hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along
>>>>>>>>>>>>>>>>>>>>> the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier
>>>>>>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the right approach
>>>>>>>>>>>>>>>>>>>>> is really about templating.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in
>>>>>>>>>>>>>>>>>>>>> a view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies
>>>>>>>>>>>>>>>>>>>>>> I shared regarding many resolution methods not consistently being followed
>>>>>>>>>>>>>>>>>>>>>> in different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback
>>>>>>>>>>>>>>>>>>>>>>> should be the catalog
>>>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter
>>>>>>>>>>>>>>>>>>>>>>> what the catalog
>>>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same
>>>>>>>>>>>>>>>>>>>>>>> catalog, this is a
>>>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog,
>>>>>>>>>>>>>>>>>>>>>>> catalog names
>>>>>>>>>>>>>>>>>>>>>>> should be included in the reference name already. So
>>>>>>>>>>>>>>>>>>>>>>> no ambiguity
>>>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a
>>>>>>>>>>>>>>>>>>>>>>> separate problem, which
>>>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage
>>>>>>>>>>>>>>>>>>>>>>> for better
>>>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the
>>>>>>>>>>>>>>>>>>>>>>> current fallback
>>>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue
>>>>>>>>>>>>>>>>>>>>>>> for not conforming
>>>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want
>>>>>>>>>>>>>>>>>>>>>>> to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at
>>>>>>>>>>>>>>>>>>>>>>> view creation time. It would be circular: the view needs to exist before
>>>>>>>>>>>>>>>>>>>>>>> its metadata (and hence default-catalog) can exist. This is visible in
>>>>>>>>>>>>>>>>>>>>>>> Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog
>>>>>>>>>>>>>>>>>>>>>>> setting would require extending SQL syntax and engine APIs to promote it to
>>>>>>>>>>>>>>>>>>>>>>> a first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the
>>>>>>>>>>>>>>>>>>>>>>> "view catalog" where the view is going to be created in. It can use that
>>>>>>>>>>>>>>>>>>>>>>> catalog to resolve partial table identifiers if "default-catalog" is not
>>>>>>>>>>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where
>>>>>>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA",
>>>>>>>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same as just
>>>>>>>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the
>>>>>>>>>>>>>>>>>>>>>>> tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How
>>>>>>>>>>>>>>>>>>>>>>> do we validate that a table exists without a default catalog, or do we
>>>>>>>>>>>>>>>>>>>>>>> always use the current session catalog?
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding.
>>>>>>>>>>>>>>>>>>>>>>> Just to clarify: when I referred to late binding as “similar” to the
>>>>>>>>>>>>>>>>>>>>>>> proposal, I was acknowledging the distinction between view-level and
>>>>>>>>>>>>>>>>>>>>>>> table-level resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view
>>>>>>>>>>>>>>>>>>>>>>> catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to
>>>>>>>>>>>>>>>>>>>>>>> distinguish between the physical catalog and it's reference used in SQL
>>>>>>>>>>>>>>>>>>>>>>> statements. The important part is that the physical catalog of the view and
>>>>>>>>>>>>>>>>>>>>>>> the tables referenced in it's definition stay consistent. You could create
>>>>>>>>>>>>>>>>>>>>>>> a view in a given physical catalog by referring to it as "catalogA", as in
>>>>>>>>>>>>>>>>>>>>>>> your first point. If you then, given a different setup, refer to the same
>>>>>>>>>>>>>>>>>>>>>>> physical catalog as "catalogB" in another session/environment, the behavior
>>>>>>>>>>>>>>>>>>>>>>> should still work.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late
>>>>>>>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension to all partial
>>>>>>>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. Resolving the view
>>>>>>>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the view metadata in a
>>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you
>>>>>>>>>>>>>>>>>>>>>>> mean.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being
>>>>>>>>>>>>>>>>>>>>>>> referenced in a SQL statement. That statement contains the information for
>>>>>>>>>>>>>>>>>>>>>>> the query engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT *
>>>>>>>>>>>>>>>>>>>>>>> FROM catalogB.namespace.view (after renaming catalogA to catalogB, and
>>>>>>>>>>>>>>>>>>>>>>> keeping everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it
>>>>>>>>>>>>>>>>>>>>>>> sounds to me like the catalog is resolved at query time, based on how the
>>>>>>>>>>>>>>>>>>>>>>> view is referenced, not from any stored metadata. That would imply some
>>>>>>>>>>>>>>>>>>>>>>> sort of a late binding behavior (similar to the proposal), as opposed to
>>>>>>>>>>>>>>>>>>>>>>> using some catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current
>>>>>>>>>>>>>>>>>>>>>>> spec: The view is executed when it's being referenced in a SQL statement.
>>>>>>>>>>>>>>>>>>>>>>> That statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can
>>>>>>>>>>>>>>>>>>>>>>> be referenced at execution time. As long as that's the case I would assume
>>>>>>>>>>>>>>>>>>>>>>> the view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because
>>>>>>>>>>>>>>>>>>>>>>> the current specification can lead to some unintuitive behavior. For
>>>>>>>>>>>>>>>>>>>>>>> example for the following statement:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view
>>>>>>>>>>>>>>>>>>>>>>> spec, because it has the "closure" property. Because of the fact that the
>>>>>>>>>>>>>>>>>>>>>>> "view catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point
>>>>>>>>>>>>>>>>>>>>>>> before going deeper:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you
>>>>>>>>>>>>>>>>>>>>>>> help clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog”
>>>>>>>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if default-catalog is
>>>>>>>>>>>>>>>>>>>>>>> null?
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we
>>>>>>>>>>>>>>>>>>>>>>> can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the
>>>>>>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of "non-determinism".
>>>>>>>>>>>>>>>>>>>>>>> You could be running the same query-engine and catalog-setup on different
>>>>>>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is rather common), and
>>>>>>>>>>>>>>>>>>>>>>> would be getting different results.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view
>>>>>>>>>>>>>>>>>>>>>>> always produces the same results. The current behavior has some rough edges
>>>>>>>>>>>>>>>>>>>>>>> in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the
>>>>>>>>>>>>>>>>>>>>>>> setup more robust. Which is great. However, having UUID validation still
>>>>>>>>>>>>>>>>>>>>>>> requires us to have a portable table identifier specification. Even if we
>>>>>>>>>>>>>>>>>>>>>>> have the UUIDs of the referenced tables from the view, there simply isn't
>>>>>>>>>>>>>>>>>>>>>>> an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup
>>>>>>>>>>>>>>>>>>>>>>> and suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior
>>>>>>>>>>>>>>>>>>>>>>> for "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have
>>>>>>>>>>>>>>>>>>>>>>> some concerns with resolving catalog names on the fly with query engine
>>>>>>>>>>>>>>>>>>>>>>> defined catalog names. This introduces some flexibility at first glance,
>>>>>>>>>>>>>>>>>>>>>>> but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should
>>>>>>>>>>>>>>>>>>>>>>> store resolved table uuid in view metadata, as table/view renaming may
>>>>>>>>>>>>>>>>>>>>>>> introduce errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum
>>>>>>>>>>>>>>>>>>>>>>> and closing out the MV spec as well. I’m hoping we can proceed to a vote
>>>>>>>>>>>>>>>>>>>>>>> next week.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed
>>>>>>>>>>>>>>>>>>>>>>> the comments.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that
>>>>>>>>>>>>>>>>>>>>>>> catalog names in the spec refer to logical catalogs—i.e., the first part of
>>>>>>>>>>>>>>>>>>>>>>> a three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think
>>>>>>>>>>>>>>>>>>>>>>> view portability is a very important topic for us to continue discussing as
>>>>>>>>>>>>>>>>>>>>>>> it relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this
>>>>>>>>>>>>>>>>>>>>>>> may impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem.
>>>>>>>>>>>>>>>>>>>>>>> I've added a few comments to get a better understanding of how this will
>>>>>>>>>>>>>>>>>>>>>>> look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with
>>>>>>>>>>>>>>>>>>>>>>> a few community members to better understand how the current spec is being
>>>>>>>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I
>>>>>>>>>>>>>>>>>>>>>>> look forward to your feedback and working together to move this forward so
>>>>>>>>>>>>>>>>>>>>>>> we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Steven Wu <st...@gmail.com>]

If a table referenced by the view SQL query is deleted and recreated, is
the regular view still valid? If yes, it won't be correct to use resolved
UUID to validate the name reference. On the materialized view side, we are
aligned that UUID is necessary to detect this scenario and mark the refresh
state "stale" if a referenced table is re-created.

> esolution using default-catalog and default-namespace does not provide
full determinism,

With the new proposal, the catalog and namespace depends on the engine's
current context. if the user changes the current context (catalog and
database) via SET/USE , the view SQL query may resolve to a different
table. That is probably a bigger problem.

With the current view spec, the intention is set in the view creation time.
I would rather have the view producer set the explicit intention (than the
implicit dependency on the engine context at view consumption phase).

Also the view SQL can still reference a table with a catalog name
component.  Hence the catalog naming still matters.The new proposal won't
really solve the naming problem.

On Thu, May 8, 2025 at 9:01 AM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> Hi Dan,
>
> Thanks for the clarification! I agree that UUIDs should not be stored in
> the SQL definition of the view.
>
> Just to clarify: the proposal isn't about modifying the view definition
> itself, but rather about how table identifiers are represented in Iceberg
> view metadata, which naturally depends on how those identifiers are
> resolved.
>
> For example, in order to store the "storage table identifier" in the
> metadata:
> * Should we apply the same resolution rules based on default-catalog and
> default-namespace?
> * And if that resolution method is considered deterministic enough, would
> you be comfortable not storing the UUID in the storage table identifier
> metadata?
>
> The same questions apply when storing child views or tables as part of the
> view’s lineage metadata. It would be helpful to clarify the resolution
> model and whether UUIDs should be part of that for validation or drift
> detection.
>
> Thanks,
> Walaa.
>
>
>
> On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org> wrote:
>
>> I don't think we want to include the resolved table UUIDs in the view
>> definition, but rather in the storage table state.  You can still resolve
>> whether those drift at some point, but I don't feel like it's a good idea
>> to capture data in the view that we may allow to drift if there isn't any
>> requirement that they match.  This also aligns with the identifier
>> resolution being late binding.
>>
>> -Dan
>>
>> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> Thanks Steven! So would you agree that resolution using default-catalog
>>> and default-namespace does not provide full determinism, and requires a
>>> supporting safety mechanism?
>>>
>>> Thanks,
>>> Walaa.
>>>
>>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com> wrote:
>>>
>>>> > If the current model is considered deterministic, do you think
>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>
>>>> I am fine with storing UUIDs for table identifiers in the view.
>>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>>> fail the query if mismatch.
>>>>
>>>> The UUID change doesn't really change the table identifier resolution
>>>> rule though. It is more of a safety protection.
>>>>
>>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> Hi Steven,
>>>>>
>>>>> Thanks for the reply.
>>>>>
>>>>> > I agree with Dan that we shouldn't solve catalog naming in the
>>>>> Iceberg view spec.
>>>>>
>>>>> To clarify, I don't believe the proposal is trying to solve catalog
>>>>> naming. What it’s doing is simply this:
>>>>>
>>>>> * Proposing that table names inside views resolve the same way as they
>>>>> do elsewhere (e.g., queries).
>>>>> * Adopting a model that is already widely used and supported in the
>>>>> existing ecosystem, which allows for:
>>>>>     -- Renaming catalog aliases
>>>>>     -- Swapping catalog implementations behind consistent names
>>>>>     -- Having different default catalog names across engines that
>>>>> still point to the same underlying tables
>>>>>
>>>>> These are common patterns in production data lakes. Saying Iceberg
>>>>> views cannot operate in those environments feels unrealistic. In practice,
>>>>> it means the spec breaks down in situations that users encounter regularly.
>>>>>
>>>>> > The recommendation of using engines’ current catalog and database
>>>>> can cause context-dependent resolution results.
>>>>>
>>>>> * As noted in the doc and earlier replies, fixing a catalog name
>>>>> doesn’t actually guarantee determinism either. All the failure scenarios
>>>>> above still apply even when a default-catalog is stored.
>>>>> * The current spec also allows default-catalog to be null, in which
>>>>> case it falls back to the view’s catalog, yet that catalog is determined
>>>>> based on how the view is referenced in the query, which would be considered
>>>>> non-deterministic based on the same criteria you shared.
>>>>> * The only true form of determinism here is UUID-based validation,
>>>>> which protects against silent drift in any resolution model.
>>>>>
>>>>> If the current model is considered deterministic, do you think
>>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>>> Or put another way: Would you be comfortable relying solely on
>>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>>> table, without UUID validation?
>>>>>
>>>>> +1 on involving other communities. I’m happy to help facilitate a
>>>>> cross-community discussion if we aren’t able to reach a resolution here.
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com> wrote:
>>>>>
>>>>>> I agree with Dan that we shouldn't solve catalog naming in the
>>>>>> Iceberg view spec. I am not convinced that the proposed change will make
>>>>>> the table identifier resolution more clear and portable. The recommendation
>>>>>> of using engines' current catalog and database can cause context dependent
>>>>>> resolution results, which seems non-deterministic to me.
>>>>>>
>>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>>> more inputs from the broader community on this part.
>>>>>>
>>>>>>
>>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:
>>>>>>
>>>>>>> In Spark, I believe that the USE commands sets the current catalog
>>>>>>> and namespace.  This affects both where the view is created and how
>>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>>> saving the current catalog and namespace into the view
>>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>>
>>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> > I think that's the lesser evil compared to Iceberg specifying
>>>>>>>> how engines should resolve identifiers
>>>>>>>>
>>>>>>>> I think this is also similar to the previous point. It is the other
>>>>>>>> way around. Right now the spec dictates how to resolve (through employing a
>>>>>>>> view-specific `default-catalog` field). The proposal is suggesting to get
>>>>>>>> out of this space and let engines handle it similar to how they handle all
>>>>>>>> identifiers.
>>>>>>>>
>>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>>>>>
>>>>>>>>> Benny, I think this is a misconception or miscommunication. The
>>>>>>>>> USE command has no impact on the `default-catalog` field. In fact, the
>>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>>> is not the case under the current spec.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> > there is no SQL construct today to explicitly set
>>>>>>>>>> default-catalog
>>>>>>>>>>
>>>>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>>>>
>>>>>>>>>> I generally agree with Dan about requiring consistent catalog
>>>>>>>>>> names.  I think that's the lesser evil compared to Iceberg specifying how
>>>>>>>>>> engines should resolve identifiers.  Another thing to consider is that
>>>>>>>>>> identifier resolution can be very expensive at query validation time if
>>>>>>>>>> identifiers need to be looked up from a bunch of places.  Hopefully, it
>>>>>>>>>> should be possible to define a view in such a way that identifiers can be
>>>>>>>>>> resolved on the first try.
>>>>>>>>>>
>>>>>>>>>> Benny
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>>
>>>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>>> materialized views.
>>>>>>>>>>>
>>>>>>>>>>> We may be able to converge on this storage format even if we
>>>>>>>>>>> haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>>> identifiers in this way.
>>>>>>>>>>>
>>>>>>>>>>> I'm supportive of moving forward with consensus on the
>>>>>>>>>>> identifier storage format. That said, we may continue to run into questions
>>>>>>>>>>> related to resolution during implementation. For example: Should the
>>>>>>>>>>> storage table identifier follow the same default-catalog and
>>>>>>>>>>> default-namespace resolution behavior as other table references?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Walaa.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>>
>>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>>
>>>>>>>>>>>> We can continue having the discussion if the current view spec
>>>>>>>>>>>> needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Rishabh
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Correction of typo: both engines seem to set default-catalog
>>>>>>>>>>>>> to the view catalog if it is defined, or to null if the view catalog is not
>>>>>>>>>>>>> defined.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree that catalog renaming is an environmental event, but
>>>>>>>>>>>>>> it's a real one that happens frequently in practice.
>>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something as
>>>>>>>>>>>>>> common as catalog renaming feels very restrictive, and could make the spec
>>>>>>>>>>>>>> less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Further, it's worth noting that the table spec is resilient
>>>>>>>>>>>>>> to catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>>> Both specifications are deterministic in their definition,
>>>>>>>>>>>>>> but one is more fragile to environmental changes than the other. Improving
>>>>>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>>>>>>> more portable over time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>>> Today, both engines seem to set default-catalog to null if
>>>>>>>>>>>>>> the view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This problem stems from the exact
>>>>>>>>>>>>>>> environmental/configuration issue that we should not be trying to address.
>>>>>>>>>>>>>>> I don't think we would expect references to survive a catalog rename.
>>>>>>>>>>>>>>> That's not something covered by the spec and needs to be handled separately
>>>>>>>>>>>>>>> as a platform-level migration specific to the affected environment.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The identifier resolution logic is clear and deterministic.
>>>>>>>>>>>>>>> It should not matter whether an engine resolves and encodes the
>>>>>>>>>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but rather
>>>>>>>>>>>>>>> view behavior when you start altering the environment around it, which
>>>>>>>>>>>>>>> isn't something we should be trying to define here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just
>>>>>>>>>>>>>>>> catalog naming consistency. The behavior around default-catalog itself
>>>>>>>>>>>>>>>> introduces resolution inconsistencies even when catalog names are
>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside views
>>>>>>>>>>>>>>>> remain reachable after a catalog rename. But if it is set to a non-null
>>>>>>>>>>>>>>>> value, table references will break.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * default-catalog causes table references inside views to
>>>>>>>>>>>>>>>> be early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the
>>>>>>>>>>>>>>>> spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg
>>>>>>>>>>>>>>>> with that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities would
>>>>>>>>>>>>>>>> align around having table resolution behave inconsistently between queries
>>>>>>>>>>>>>>>> and views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog'
>>>>>>>>>>>>>>>>> was introduced, but I think we need to step back and align on what we are
>>>>>>>>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent to
>>>>>>>>>>>>>>>>> address the catalog naming/configuration and would argue that we shouldn't
>>>>>>>>>>>>>>>>> attempt to do that as part of the spec.  Inconsistently named catalogs are
>>>>>>>>>>>>>>>>> a reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We should support and advocate for consistency in catalog
>>>>>>>>>>>>>>>>> naming and define the spec along those lines.  The fact is that with all of
>>>>>>>>>>>>>>>>> the recent work that's gone into making catalogs pluggable, it makes more
>>>>>>>>>>>>>>>>> sense to just register catalog configuration with consistent names (even if
>>>>>>>>>>>>>>>>> you have to duplicate the configuration for supporting existing
>>>>>>>>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late
>>>>>>>>>>>>>>>>> binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was that
>>>>>>>>>>>>>>>>>> every query engine uses it to store its session default catalog at the time
>>>>>>>>>>>>>>>>>> of creating the view. This way the view could be reused in another session.
>>>>>>>>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Generally we have different environments we want to
>>>>>>>>>>>>>>>>>> support with the view spec:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When the environment supports it, using consistent
>>>>>>>>>>>>>>>>>> catalog names can have a great benefit for multi-catalog, multi-engine
>>>>>>>>>>>>>>>>>> setups. With consistent catalog names, using the "default-catalog" field
>>>>>>>>>>>>>>>>>> works without any issues.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This can be the case when different query engines refer
>>>>>>>>>>>>>>>>>> to the same physical catalog by different names. This often happens because
>>>>>>>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields
>>>>>>>>>>>>>>>>>> from the view spec completely.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level
>>>>>>>>>>>>>>>>>> default catalogs or namespaces.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as
>>>>>>>>>>>>>>>>>> they appear in the view SQL text.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it
>>>>>>>>>>>>>>>>>> should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion
>>>>>>>>>>>>>>>>>>> the fallback creates. Also just to be clear, at this point and after
>>>>>>>>>>>>>>>>>>> clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't
>>>>>>>>>>>>>>>>>>>> think we should design something that says first use this catalog, then
>>>>>>>>>>>>>>>>>>>> that catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see
>>>>>>>>>>>>>>>>>>>> no value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>>> - Interoperability between different engine types is
>>>>>>>>>>>>>>>>>>>> very hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along
>>>>>>>>>>>>>>>>>>>> the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier
>>>>>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the right approach
>>>>>>>>>>>>>>>>>>>> is really about templating.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a
>>>>>>>>>>>>>>>>>>>> view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I
>>>>>>>>>>>>>>>>>>>>> shared regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback
>>>>>>>>>>>>>>>>>>>>>> should be the catalog
>>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter
>>>>>>>>>>>>>>>>>>>>>> what the catalog
>>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog,
>>>>>>>>>>>>>>>>>>>>>> this is a
>>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog,
>>>>>>>>>>>>>>>>>>>>>> catalog names
>>>>>>>>>>>>>>>>>>>>>> should be included in the reference name already. So
>>>>>>>>>>>>>>>>>>>>>> no ambiguity
>>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a
>>>>>>>>>>>>>>>>>>>>>> separate problem, which
>>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage for
>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the
>>>>>>>>>>>>>>>>>>>>>> current fallback
>>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue
>>>>>>>>>>>>>>>>>>>>>> for not conforming
>>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want
>>>>>>>>>>>>>>>>>>>>>> to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at
>>>>>>>>>>>>>>>>>>>>>> view creation time. It would be circular: the view needs to exist before
>>>>>>>>>>>>>>>>>>>>>> its metadata (and hence default-catalog) can exist. This is visible in
>>>>>>>>>>>>>>>>>>>>>> Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog
>>>>>>>>>>>>>>>>>>>>>> setting would require extending SQL syntax and engine APIs to promote it to
>>>>>>>>>>>>>>>>>>>>>> a first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the
>>>>>>>>>>>>>>>>>>>>>> "view catalog" where the view is going to be created in. It can use that
>>>>>>>>>>>>>>>>>>>>>> catalog to resolve partial table identifiers if "default-catalog" is not
>>>>>>>>>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where
>>>>>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA",
>>>>>>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same as just
>>>>>>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the
>>>>>>>>>>>>>>>>>>>>>> tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do
>>>>>>>>>>>>>>>>>>>>>> we validate that a table exists without a default catalog, or do we always
>>>>>>>>>>>>>>>>>>>>>> use the current session catalog?
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding.
>>>>>>>>>>>>>>>>>>>>>> Just to clarify: when I referred to late binding as “similar” to the
>>>>>>>>>>>>>>>>>>>>>> proposal, I was acknowledging the distinction between view-level and
>>>>>>>>>>>>>>>>>>>>>> table-level resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view
>>>>>>>>>>>>>>>>>>>>>> catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to
>>>>>>>>>>>>>>>>>>>>>> distinguish between the physical catalog and it's reference used in SQL
>>>>>>>>>>>>>>>>>>>>>> statements. The important part is that the physical catalog of the view and
>>>>>>>>>>>>>>>>>>>>>> the tables referenced in it's definition stay consistent. You could create
>>>>>>>>>>>>>>>>>>>>>> a view in a given physical catalog by referring to it as "catalogA", as in
>>>>>>>>>>>>>>>>>>>>>> your first point. If you then, given a different setup, refer to the same
>>>>>>>>>>>>>>>>>>>>>> physical catalog as "catalogB" in another session/environment, the behavior
>>>>>>>>>>>>>>>>>>>>>> should still work.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late
>>>>>>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension to all partial
>>>>>>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. Resolving the view
>>>>>>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the view metadata in a
>>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you
>>>>>>>>>>>>>>>>>>>>>> mean.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being
>>>>>>>>>>>>>>>>>>>>>> referenced in a SQL statement. That statement contains the information for
>>>>>>>>>>>>>>>>>>>>>> the query engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT *
>>>>>>>>>>>>>>>>>>>>>> FROM catalogB.namespace.view (after renaming catalogA to catalogB, and
>>>>>>>>>>>>>>>>>>>>>> keeping everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds
>>>>>>>>>>>>>>>>>>>>>> to me like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current
>>>>>>>>>>>>>>>>>>>>>> spec: The view is executed when it's being referenced in a SQL statement.
>>>>>>>>>>>>>>>>>>>>>> That statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can
>>>>>>>>>>>>>>>>>>>>>> be referenced at execution time. As long as that's the case I would assume
>>>>>>>>>>>>>>>>>>>>>> the view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because
>>>>>>>>>>>>>>>>>>>>>> the current specification can lead to some unintuitive behavior. For
>>>>>>>>>>>>>>>>>>>>>> example for the following statement:
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view
>>>>>>>>>>>>>>>>>>>>>> spec, because it has the "closure" property. Because of the fact that the
>>>>>>>>>>>>>>>>>>>>>> "view catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point
>>>>>>>>>>>>>>>>>>>>>> before going deeper:
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you
>>>>>>>>>>>>>>>>>>>>>> help clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog”
>>>>>>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if default-catalog is
>>>>>>>>>>>>>>>>>>>>>> null?
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we
>>>>>>>>>>>>>>>>>>>>>> can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the
>>>>>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of "non-determinism".
>>>>>>>>>>>>>>>>>>>>>> You could be running the same query-engine and catalog-setup on different
>>>>>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is rather common), and
>>>>>>>>>>>>>>>>>>>>>> would be getting different results.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view
>>>>>>>>>>>>>>>>>>>>>> always produces the same results. The current behavior has some rough edges
>>>>>>>>>>>>>>>>>>>>>> in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the
>>>>>>>>>>>>>>>>>>>>>> setup more robust. Which is great. However, having UUID validation still
>>>>>>>>>>>>>>>>>>>>>> requires us to have a portable table identifier specification. Even if we
>>>>>>>>>>>>>>>>>>>>>> have the UUIDs of the referenced tables from the view, there simply isn't
>>>>>>>>>>>>>>>>>>>>>> an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup
>>>>>>>>>>>>>>>>>>>>>> and suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior
>>>>>>>>>>>>>>>>>>>>>> for "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have
>>>>>>>>>>>>>>>>>>>>>> some concerns with resolving catalog names on the fly with query engine
>>>>>>>>>>>>>>>>>>>>>> defined catalog names. This introduces some flexibility at first glance,
>>>>>>>>>>>>>>>>>>>>>> but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should
>>>>>>>>>>>>>>>>>>>>>> store resolved table uuid in view metadata, as table/view renaming may
>>>>>>>>>>>>>>>>>>>>>> introduce errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum
>>>>>>>>>>>>>>>>>>>>>> and closing out the MV spec as well. I’m hoping we can proceed to a vote
>>>>>>>>>>>>>>>>>>>>>> next week.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed
>>>>>>>>>>>>>>>>>>>>>> the comments.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that
>>>>>>>>>>>>>>>>>>>>>> catalog names in the spec refer to logical catalogs—i.e., the first part of
>>>>>>>>>>>>>>>>>>>>>> a three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think
>>>>>>>>>>>>>>>>>>>>>> view portability is a very important topic for us to continue discussing as
>>>>>>>>>>>>>>>>>>>>>> it relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this
>>>>>>>>>>>>>>>>>>>>>> may impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem.
>>>>>>>>>>>>>>>>>>>>>> I've added a few comments to get a better understanding of how this will
>>>>>>>>>>>>>>>>>>>>>> look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa
>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a
>>>>>>>>>>>>>>>>>>>>>> few community members to better understand how the current spec is being
>>>>>>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I
>>>>>>>>>>>>>>>>>>>>>> look forward to your feedback and working together to move this forward so
>>>>>>>>>>>>>>>>>>>>>> we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

Hi Dan,

Thanks for the clarification! I agree that UUIDs should not be stored in
the SQL definition of the view.

Just to clarify: the proposal isn't about modifying the view definition
itself, but rather about how table identifiers are represented in Iceberg
view metadata, which naturally depends on how those identifiers are
resolved.

For example, in order to store the "storage table identifier" in the
metadata:
* Should we apply the same resolution rules based on default-catalog and
default-namespace?
* And if that resolution method is considered deterministic enough, would
you be comfortable not storing the UUID in the storage table identifier
metadata?

The same questions apply when storing child views or tables as part of the
view’s lineage metadata. It would be helpful to clarify the resolution
model and whether UUIDs should be part of that for validation or drift
detection.

Thanks,
Walaa.



On Thu, May 8, 2025 at 8:29 AM Daniel Weeks <dw...@apache.org> wrote:

> I don't think we want to include the resolved table UUIDs in the view
> definition, but rather in the storage table state.  You can still resolve
> whether those drift at some point, but I don't feel like it's a good idea
> to capture data in the view that we may allow to drift if there isn't any
> requirement that they match.  This also aligns with the identifier
> resolution being late binding.
>
> -Dan
>
> On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> Thanks Steven! So would you agree that resolution using default-catalog
>> and default-namespace does not provide full determinism, and requires a
>> supporting safety mechanism?
>>
>> Thanks,
>> Walaa.
>>
>> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com> wrote:
>>
>>> > If the current model is considered deterministic, do you think
>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>> to eliminate the need for UUIDs when storing table identifiers?
>>>
>>> I am fine with storing UUIDs for table identifiers in the view.
>>> Basically, view creation resolves all referenced tables/views with UUIDs.
>>> View consumers can validate resolved tables/views with the stored UUIDs and
>>> fail the query if mismatch.
>>>
>>> The UUID change doesn't really change the table identifier resolution
>>> rule though. It is more of a safety protection.
>>>
>>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> Hi Steven,
>>>>
>>>> Thanks for the reply.
>>>>
>>>> > I agree with Dan that we shouldn't solve catalog naming in the
>>>> Iceberg view spec.
>>>>
>>>> To clarify, I don't believe the proposal is trying to solve catalog
>>>> naming. What it’s doing is simply this:
>>>>
>>>> * Proposing that table names inside views resolve the same way as they
>>>> do elsewhere (e.g., queries).
>>>> * Adopting a model that is already widely used and supported in the
>>>> existing ecosystem, which allows for:
>>>>     -- Renaming catalog aliases
>>>>     -- Swapping catalog implementations behind consistent names
>>>>     -- Having different default catalog names across engines that still
>>>> point to the same underlying tables
>>>>
>>>> These are common patterns in production data lakes. Saying Iceberg
>>>> views cannot operate in those environments feels unrealistic. In practice,
>>>> it means the spec breaks down in situations that users encounter regularly.
>>>>
>>>> > The recommendation of using engines’ current catalog and database can
>>>> cause context-dependent resolution results.
>>>>
>>>> * As noted in the doc and earlier replies, fixing a catalog name
>>>> doesn’t actually guarantee determinism either. All the failure scenarios
>>>> above still apply even when a default-catalog is stored.
>>>> * The current spec also allows default-catalog to be null, in which
>>>> case it falls back to the view’s catalog, yet that catalog is determined
>>>> based on how the view is referenced in the query, which would be considered
>>>> non-deterministic based on the same criteria you shared.
>>>> * The only true form of determinism here is UUID-based validation,
>>>> which protects against silent drift in any resolution model.
>>>>
>>>> If the current model is considered deterministic, do you think
>>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>>> to eliminate the need for UUIDs when storing table identifiers?
>>>> Or put another way: Would you be comfortable relying solely on
>>>> default-catalog + default-namespace + table name to re-identify the correct
>>>> table, without UUID validation?
>>>>
>>>> +1 on involving other communities. I’m happy to help facilitate a
>>>> cross-community discussion if we aren’t able to reach a resolution here.
>>>>
>>>> Thanks,
>>>> Walaa.
>>>>
>>>>
>>>>
>>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com> wrote:
>>>>
>>>>> I agree with Dan that we shouldn't solve catalog naming in the Iceberg
>>>>> view spec. I am not convinced that the proposed change will make the table
>>>>> identifier resolution more clear and portable. The recommendation of using
>>>>> engines' current catalog and database can cause context dependent
>>>>> resolution results, which seems non-deterministic to me.
>>>>>
>>>>> Walaa, you raised a point in the doc that the current catalog
>>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>>> more inputs from the broader community on this part.
>>>>>
>>>>>
>>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:
>>>>>
>>>>>> In Spark, I believe that the USE commands sets the current catalog
>>>>>> and namespace.  This affects both where the view is created and how
>>>>>> unqualified table identifiers are resolved.  I also don't see an issue with
>>>>>> saving the current catalog and namespace into the view
>>>>>> metadata's default-catalog and default-namespace fields.
>>>>>>
>>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> > I think that's the lesser evil compared to Iceberg specifying how
>>>>>>> engines should resolve identifiers
>>>>>>>
>>>>>>> I think this is also similar to the previous point. It is the other
>>>>>>> way around. Right now the spec dictates how to resolve (through employing a
>>>>>>> view-specific `default-catalog` field). The proposal is suggesting to get
>>>>>>> out of this space and let engines handle it similar to how they handle all
>>>>>>> identifiers.
>>>>>>>
>>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>>>>
>>>>>>>> Benny, I think this is a misconception or miscommunication. The USE
>>>>>>>> command has no impact on the `default-catalog` field. In fact, the
>>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>>> is not the case under the current spec.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> > there is no SQL construct today to explicitly set default-catalog
>>>>>>>>>
>>>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>>>
>>>>>>>>> I generally agree with Dan about requiring consistent catalog
>>>>>>>>> names.  I think that's the lesser evil compared to Iceberg specifying how
>>>>>>>>> engines should resolve identifiers.  Another thing to consider is that
>>>>>>>>> identifier resolution can be very expensive at query validation time if
>>>>>>>>> identifiers need to be looked up from a bunch of places.  Hopefully, it
>>>>>>>>> should be possible to define a view in such a way that identifiers can be
>>>>>>>>> resolved on the first try.
>>>>>>>>>
>>>>>>>>> Benny
>>>>>>>>>
>>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Rishabh,
>>>>>>>>>>
>>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>>> materialized views.
>>>>>>>>>>
>>>>>>>>>> We may be able to converge on this storage format even if we
>>>>>>>>>> haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>>> identifiers in this way.
>>>>>>>>>>
>>>>>>>>>> I'm supportive of moving forward with consensus on the identifier
>>>>>>>>>> storage format. That said, we may continue to run into questions related to
>>>>>>>>>> resolution during implementation. For example: Should the storage table
>>>>>>>>>> identifier follow the same default-catalog and default-namespace resolution
>>>>>>>>>> behavior as other table references?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Walaa.
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Walaa,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>>
>>>>>>>>>>> I think we should decouple at least the MV Spec from the
>>>>>>>>>>> proposal to change the current behavior of view resolution.
>>>>>>>>>>>
>>>>>>>>>>> We can continue having the discussion if the current view spec
>>>>>>>>>>> needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Rishabh
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Correction of typo: both engines seem to set default-catalog to
>>>>>>>>>>>> the view catalog if it is defined, or to null if the view catalog is not
>>>>>>>>>>>> defined.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree that catalog renaming is an environmental event, but
>>>>>>>>>>>>> it's a real one that happens frequently in practice.
>>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something as
>>>>>>>>>>>>> common as catalog renaming feels very restrictive, and could make the spec
>>>>>>>>>>>>> less practical, even unusable, for real-world deployments.
>>>>>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Further, it's worth noting that the table spec is resilient to
>>>>>>>>>>>>> catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>>> Both specifications are deterministic in their definition, but
>>>>>>>>>>>>> one is more fragile to environmental changes than the other. Improving
>>>>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>>>>>> more portable over time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>>> Today, both engines seem to set default-catalog to null if the
>>>>>>>>>>>>> view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <
>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This problem stems from the exact environmental/configuration
>>>>>>>>>>>>>> issue that we should not be trying to address.  I don't think we would
>>>>>>>>>>>>>> expect references to survive a catalog rename.  That's not something
>>>>>>>>>>>>>> covered by the spec and needs to be handled separately as a platform-level
>>>>>>>>>>>>>> migration specific to the affected environment.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The identifier resolution logic is clear and deterministic.
>>>>>>>>>>>>>> It should not matter whether an engine resolves and encodes the
>>>>>>>>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>>>>>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>>>>>>>>>> something we should be trying to define here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just catalog
>>>>>>>>>>>>>>> naming consistency. The behavior around default-catalog itself introduces
>>>>>>>>>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside views
>>>>>>>>>>>>>>> remain reachable after a catalog rename. But if it is set to a non-null
>>>>>>>>>>>>>>> value, table references will break.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * default-catalog causes table references inside views to be
>>>>>>>>>>>>>>> early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the
>>>>>>>>>>>>>>> spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with
>>>>>>>>>>>>>>> that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities would
>>>>>>>>>>>>>>> align around having table resolution behave inconsistently between queries
>>>>>>>>>>>>>>> and views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog'
>>>>>>>>>>>>>>>> was introduced, but I think we need to step back and align on what we are
>>>>>>>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The issues we're discussing largely stem from using
>>>>>>>>>>>>>>>> multiple engines with cross catalog references and configurations where
>>>>>>>>>>>>>>>> catalog names are not aligned.  If we have multiple engines that all have
>>>>>>>>>>>>>>>> the same catalog names/configurations, the current spec implementation is
>>>>>>>>>>>>>>>> well defined for table resolution even across catalogs.  The
>>>>>>>>>>>>>>>> 'default-catalog' (and namespace equivalent) was intended to address the
>>>>>>>>>>>>>>>> resolution within the context of the sql text, not to address
>>>>>>>>>>>>>>>> catalog/naming inconsistencies.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent to
>>>>>>>>>>>>>>>> address the catalog naming/configuration and would argue that we shouldn't
>>>>>>>>>>>>>>>> attempt to do that as part of the spec.  Inconsistently named catalogs are
>>>>>>>>>>>>>>>> a reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We should support and advocate for consistency in catalog
>>>>>>>>>>>>>>>> naming and define the spec along those lines.  The fact is that with all of
>>>>>>>>>>>>>>>> the recent work that's gone into making catalogs pluggable, it makes more
>>>>>>>>>>>>>>>> sense to just register catalog configuration with consistent names (even if
>>>>>>>>>>>>>>>> you have to duplicate the configuration for supporting existing
>>>>>>>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late
>>>>>>>>>>>>>>>> binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was that
>>>>>>>>>>>>>>>>> every query engine uses it to store its session default catalog at the time
>>>>>>>>>>>>>>>>> of creating the view. This way the view could be reused in another session.
>>>>>>>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Generally we have different environments we want to
>>>>>>>>>>>>>>>>> support with the view spec:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When the environment supports it, using consistent catalog
>>>>>>>>>>>>>>>>> names can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>>>>>>>>>> any issues.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This can be the case when different query engines refer to
>>>>>>>>>>>>>>>>> the same physical catalog by different names. This often happens because
>>>>>>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields from
>>>>>>>>>>>>>>>>> the view spec completely.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level
>>>>>>>>>>>>>>>>> default catalogs or namespaces.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as
>>>>>>>>>>>>>>>>> they appear in the view SQL text.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it
>>>>>>>>>>>>>>>>> should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever
>>>>>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion
>>>>>>>>>>>>>>>>>> the fallback creates. Also just to be clear, at this point and after
>>>>>>>>>>>>>>>>>> clarifying the current spec intentions, I am convinced that we should
>>>>>>>>>>>>>>>>>> remove the default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't
>>>>>>>>>>>>>>>>>>> think we should design something that says first use this catalog, then
>>>>>>>>>>>>>>>>>>> that catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no
>>>>>>>>>>>>>>>>>>> value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>>> - Interoperability between different engine types is
>>>>>>>>>>>>>>>>>>> very hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along
>>>>>>>>>>>>>>>>>>> the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier
>>>>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the right approach
>>>>>>>>>>>>>>>>>>> is really about templating.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a
>>>>>>>>>>>>>>>>>>> view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I
>>>>>>>>>>>>>>>>>>>> shared regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback
>>>>>>>>>>>>>>>>>>>>> should be the catalog
>>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter
>>>>>>>>>>>>>>>>>>>>> what the catalog
>>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog,
>>>>>>>>>>>>>>>>>>>>> this is a
>>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog,
>>>>>>>>>>>>>>>>>>>>> catalog names
>>>>>>>>>>>>>>>>>>>>> should be included in the reference name already. So
>>>>>>>>>>>>>>>>>>>>> no ambiguity
>>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage for
>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the
>>>>>>>>>>>>>>>>>>>>> current fallback
>>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for
>>>>>>>>>>>>>>>>>>>>> not conforming
>>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want
>>>>>>>>>>>>>>>>>>>>> to highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at
>>>>>>>>>>>>>>>>>>>>> view creation time. It would be circular: the view needs to exist before
>>>>>>>>>>>>>>>>>>>>> its metadata (and hence default-catalog) can exist. This is visible in
>>>>>>>>>>>>>>>>>>>>> Spark’s implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog
>>>>>>>>>>>>>>>>>>>>> setting would require extending SQL syntax and engine APIs to promote it to
>>>>>>>>>>>>>>>>>>>>> a first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from
>>>>>>>>>>>>>>>>>>>>> this idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the
>>>>>>>>>>>>>>>>>>>>> "view catalog" where the view is going to be created in. It can use that
>>>>>>>>>>>>>>>>>>>>> catalog to resolve partial table identifiers if "default-catalog" is not
>>>>>>>>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where
>>>>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT
>>>>>>>>>>>>>>>>>>>>> * from sales.orders;
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA",
>>>>>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same as just
>>>>>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the
>>>>>>>>>>>>>>>>>>>>> tables referenced in the view exist, how can we do that when
>>>>>>>>>>>>>>>>>>>>> default-catalog isn't defined, since the view hasn't been created or loaded
>>>>>>>>>>>>>>>>>>>>> yet?
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do
>>>>>>>>>>>>>>>>>>>>> we validate that a table exists without a default catalog, or do we always
>>>>>>>>>>>>>>>>>>>>> use the current session catalog?
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding.
>>>>>>>>>>>>>>>>>>>>> Just to clarify: when I referred to late binding as “similar” to the
>>>>>>>>>>>>>>>>>>>>> proposal, I was acknowledging the distinction between view-level and
>>>>>>>>>>>>>>>>>>>>> table-level resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view
>>>>>>>>>>>>>>>>>>>>> catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish
>>>>>>>>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>>>>>>>>>> still work.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late
>>>>>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension to all partial
>>>>>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. Resolving the view
>>>>>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the view metadata in a
>>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you
>>>>>>>>>>>>>>>>>>>>> mean.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced
>>>>>>>>>>>>>>>>>>>>> in a SQL statement. That statement contains the information for the query
>>>>>>>>>>>>>>>>>>>>> engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT *
>>>>>>>>>>>>>>>>>>>>> FROM catalogB.namespace.view (after renaming catalogA to catalogB, and
>>>>>>>>>>>>>>>>>>>>> keeping everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds
>>>>>>>>>>>>>>>>>>>>> to me like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current
>>>>>>>>>>>>>>>>>>>>> spec: The view is executed when it's being referenced in a SQL statement.
>>>>>>>>>>>>>>>>>>>>> That statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because
>>>>>>>>>>>>>>>>>>>>> the current specification can lead to some unintuitive behavior. For
>>>>>>>>>>>>>>>>>>>>> example for the following statement:
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not
>>>>>>>>>>>>>>>>>>>>> "catalogA", the "sales.orders" in the view query would not be the same as
>>>>>>>>>>>>>>>>>>>>> just referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view
>>>>>>>>>>>>>>>>>>>>> spec, because it has the "closure" property. Because of the fact that the
>>>>>>>>>>>>>>>>>>>>> "view catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point
>>>>>>>>>>>>>>>>>>>>> before going deeper:
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session
>>>>>>>>>>>>>>>>>>>>> fallback behavior—it’s a fundamental limitation of using table identifiers
>>>>>>>>>>>>>>>>>>>>> alone, regardless of whether we use the current rule, the proposed fallback
>>>>>>>>>>>>>>>>>>>>> to the session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help
>>>>>>>>>>>>>>>>>>>>> clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog”
>>>>>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if default-catalog is
>>>>>>>>>>>>>>>>>>>>> null?
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we
>>>>>>>>>>>>>>>>>>>>> can’t unambiguously recover the "view catalog" from metadata, then
>>>>>>>>>>>>>>>>>>>>> defaulting to it is problematic. And if views can't be created in the
>>>>>>>>>>>>>>>>>>>>> default catalog, then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the
>>>>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of "non-determinism".
>>>>>>>>>>>>>>>>>>>>> You could be running the same query-engine and catalog-setup on different
>>>>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is rather common), and
>>>>>>>>>>>>>>>>>>>>> would be getting different results.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view
>>>>>>>>>>>>>>>>>>>>> always produces the same results. The current behavior has some rough edges
>>>>>>>>>>>>>>>>>>>>> in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the
>>>>>>>>>>>>>>>>>>>>> setup more robust. Which is great. However, having UUID validation still
>>>>>>>>>>>>>>>>>>>>> requires us to have a portable table identifier specification. Even if we
>>>>>>>>>>>>>>>>>>>>> have the UUIDs of the referenced tables from the view, there simply isn't
>>>>>>>>>>>>>>>>>>>>> an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup
>>>>>>>>>>>>>>>>>>>>> and suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior
>>>>>>>>>>>>>>>>>>>>> for "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on
>>>>>>>>>>>>>>>>>>>>> resolving catalogs on the fly already (to address the case of view text
>>>>>>>>>>>>>>>>>>>>> with table identifiers missing the catalog part). The guidance is to use
>>>>>>>>>>>>>>>>>>>>> the catalog where the view is stored. But I find this rule hard to
>>>>>>>>>>>>>>>>>>>>> interpret or use. The catalog itself is a logical construct—such as a
>>>>>>>>>>>>>>>>>>>>> federated catalog that delegates to multiple physical backends (e.g., HMS
>>>>>>>>>>>>>>>>>>>>> and REST). In such cases, the catalog (e.g., `my_catalog` in
>>>>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) doesn’t physically store the tables; it
>>>>>>>>>>>>>>>>>>>>> only routes requests to underlying stores. Therefore, defaulting identifier
>>>>>>>>>>>>>>>>>>>>> resolution based on the catalog where the view is "stored" doesn’t align
>>>>>>>>>>>>>>>>>>>>> with how catalogs actually behave in practice.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have
>>>>>>>>>>>>>>>>>>>>> some concerns with resolving catalog names on the fly with query engine
>>>>>>>>>>>>>>>>>>>>> defined catalog names. This introduces some flexibility at first glance,
>>>>>>>>>>>>>>>>>>>>> but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum
>>>>>>>>>>>>>>>>>>>>> and closing out the MV spec as well. I’m hoping we can proceed to a vote
>>>>>>>>>>>>>>>>>>>>> next week.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that
>>>>>>>>>>>>>>>>>>>>> catalog names in the spec refer to logical catalogs—i.e., the first part of
>>>>>>>>>>>>>>>>>>>>> a three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think
>>>>>>>>>>>>>>>>>>>>> view portability is a very important topic for us to continue discussing as
>>>>>>>>>>>>>>>>>>>>> it relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this
>>>>>>>>>>>>>>>>>>>>> may impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem.
>>>>>>>>>>>>>>>>>>>>> I've added a few comments to get a better understanding of how this will
>>>>>>>>>>>>>>>>>>>>> look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa
>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a
>>>>>>>>>>>>>>>>>>>>> few community members to better understand how the current spec is being
>>>>>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I
>>>>>>>>>>>>>>>>>>>>> look forward to your feedback and working together to move this forward so
>>>>>>>>>>>>>>>>>>>>> we can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Daniel Weeks <dw...@apache.org>]

I don't think we want to include the resolved table UUIDs in the view
definition, but rather in the storage table state.  You can still resolve
whether those drift at some point, but I don't feel like it's a good idea
to capture data in the view that we may allow to drift if there isn't any
requirement that they match.  This also aligns with the identifier
resolution being late binding.

-Dan

On Wed, May 7, 2025 at 10:45 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> Thanks Steven! So would you agree that resolution using default-catalog
> and default-namespace does not provide full determinism, and requires a
> supporting safety mechanism?
>
> Thanks,
> Walaa.
>
> On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com> wrote:
>
>> > If the current model is considered deterministic, do you think
>> `default-catalog` and `default-namespace` fields provide enough determinism
>> to eliminate the need for UUIDs when storing table identifiers?
>>
>> I am fine with storing UUIDs for table identifiers in the view.
>> Basically, view creation resolves all referenced tables/views with UUIDs.
>> View consumers can validate resolved tables/views with the stored UUIDs and
>> fail the query if mismatch.
>>
>> The UUID change doesn't really change the table identifier resolution
>> rule though. It is more of a safety protection.
>>
>> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> Hi Steven,
>>>
>>> Thanks for the reply.
>>>
>>> > I agree with Dan that we shouldn't solve catalog naming in the Iceberg
>>> view spec.
>>>
>>> To clarify, I don't believe the proposal is trying to solve catalog
>>> naming. What it’s doing is simply this:
>>>
>>> * Proposing that table names inside views resolve the same way as they
>>> do elsewhere (e.g., queries).
>>> * Adopting a model that is already widely used and supported in the
>>> existing ecosystem, which allows for:
>>>     -- Renaming catalog aliases
>>>     -- Swapping catalog implementations behind consistent names
>>>     -- Having different default catalog names across engines that still
>>> point to the same underlying tables
>>>
>>> These are common patterns in production data lakes. Saying Iceberg views
>>> cannot operate in those environments feels unrealistic. In practice, it
>>> means the spec breaks down in situations that users encounter regularly.
>>>
>>> > The recommendation of using engines’ current catalog and database can
>>> cause context-dependent resolution results.
>>>
>>> * As noted in the doc and earlier replies, fixing a catalog name doesn’t
>>> actually guarantee determinism either. All the failure scenarios above
>>> still apply even when a default-catalog is stored.
>>> * The current spec also allows default-catalog to be null, in which case
>>> it falls back to the view’s catalog, yet that catalog is determined based
>>> on how the view is referenced in the query, which would be considered
>>> non-deterministic based on the same criteria you shared.
>>> * The only true form of determinism here is UUID-based validation, which
>>> protects against silent drift in any resolution model.
>>>
>>> If the current model is considered deterministic, do you think
>>> `default-catalog` and `default-namespace` fields provide enough determinism
>>> to eliminate the need for UUIDs when storing table identifiers?
>>> Or put another way: Would you be comfortable relying solely on
>>> default-catalog + default-namespace + table name to re-identify the correct
>>> table, without UUID validation?
>>>
>>> +1 on involving other communities. I’m happy to help facilitate a
>>> cross-community discussion if we aren’t able to reach a resolution here.
>>>
>>> Thanks,
>>> Walaa.
>>>
>>>
>>>
>>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com> wrote:
>>>
>>>> I agree with Dan that we shouldn't solve catalog naming in the Iceberg
>>>> view spec. I am not convinced that the proposed change will make the table
>>>> identifier resolution more clear and portable. The recommendation of using
>>>> engines' current catalog and database can cause context dependent
>>>> resolution results, which seems non-deterministic to me.
>>>>
>>>> Walaa, you raised a point in the doc that the current catalog
>>>> resolution logic (default-catalog field, then view catalog) is challenging
>>>> and unrealistic for engines (like Spark and Trino). It will be great to get
>>>> more inputs from the broader community on this part.
>>>>
>>>>
>>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:
>>>>
>>>>> In Spark, I believe that the USE commands sets the current catalog and
>>>>> namespace.  This affects both where the view is created and how unqualified
>>>>> table identifiers are resolved.  I also don't see an issue with saving the
>>>>> current catalog and namespace into the view metadata's default-catalog and
>>>>> default-namespace fields.
>>>>>
>>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> > I think that's the lesser evil compared to Iceberg specifying how
>>>>>> engines should resolve identifiers
>>>>>>
>>>>>> I think this is also similar to the previous point. It is the other
>>>>>> way around. Right now the spec dictates how to resolve (through employing a
>>>>>> view-specific `default-catalog` field). The proposal is suggesting to get
>>>>>> out of this space and let engines handle it similar to how they handle all
>>>>>> identifiers.
>>>>>>
>>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>>>
>>>>>>> Benny, I think this is a misconception or miscommunication. The USE
>>>>>>> command has no impact on the `default-catalog` field. In fact, the
>>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>>> is not the case under the current spec.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com> wrote:
>>>>>>>
>>>>>>>> > there is no SQL construct today to explicitly set default-catalog
>>>>>>>>
>>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>>
>>>>>>>> I generally agree with Dan about requiring consistent catalog
>>>>>>>> names.  I think that's the lesser evil compared to Iceberg specifying how
>>>>>>>> engines should resolve identifiers.  Another thing to consider is that
>>>>>>>> identifier resolution can be very expensive at query validation time if
>>>>>>>> identifiers need to be looked up from a bunch of places.  Hopefully, it
>>>>>>>> should be possible to define a view in such a way that identifiers can be
>>>>>>>> resolved on the first try.
>>>>>>>>
>>>>>>>> Benny
>>>>>>>>
>>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Rishabh,
>>>>>>>>>
>>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>>> materialized views.
>>>>>>>>>
>>>>>>>>> We may be able to converge on this storage format even if we
>>>>>>>>> haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>>> identifiers in this way.
>>>>>>>>>
>>>>>>>>> I'm supportive of moving forward with consensus on the identifier
>>>>>>>>> storage format. That said, we may continue to run into questions related to
>>>>>>>>> resolution during implementation. For example: Should the storage table
>>>>>>>>> identifier follow the same default-catalog and default-namespace resolution
>>>>>>>>> behavior as other table references?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Walaa.
>>>>>>>>>
>>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Walaa,
>>>>>>>>>>
>>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>>
>>>>>>>>>> I think we should decouple at least the MV Spec from the proposal
>>>>>>>>>> to change the current behavior of view resolution.
>>>>>>>>>>
>>>>>>>>>> We can continue having the discussion if the current view spec
>>>>>>>>>> needs to be changed or not. Based on the decision at a later point if
>>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Rishabh
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Correction of typo: both engines seem to set default-catalog to
>>>>>>>>>>> the view catalog if it is defined, or to null if the view catalog is not
>>>>>>>>>>> defined.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>>
>>>>>>>>>>>> I agree that catalog renaming is an environmental event, but
>>>>>>>>>>>> it's a real one that happens frequently in practice.
>>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something as
>>>>>>>>>>>> common as catalog renaming feels very restrictive, and could make the spec
>>>>>>>>>>>> less practical, even unusable, for real-world deployments.
>>>>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>>>>
>>>>>>>>>>>> Further, it's worth noting that the table spec is resilient to
>>>>>>>>>>>> catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>>> Both specifications are deterministic in their definition, but
>>>>>>>>>>>> one is more fragile to environmental changes than the other. Improving
>>>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>>>>> more portable over time.
>>>>>>>>>>>>
>>>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>>> Today, both engines seem to set default-catalog to null if the
>>>>>>>>>>>> view catalog is defined, or to the view catalog if not.
>>>>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Walaa
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dw...@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>>
>>>>>>>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>>>>>>>
>>>>>>>>>>>>> This problem stems from the exact environmental/configuration
>>>>>>>>>>>>> issue that we should not be trying to address.  I don't think we would
>>>>>>>>>>>>> expect references to survive a catalog rename.  That's not something
>>>>>>>>>>>>> covered by the spec and needs to be handled separately as a platform-level
>>>>>>>>>>>>> migration specific to the affected environment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The identifier resolution logic is clear and deterministic.
>>>>>>>>>>>>> It should not matter whether an engine resolves and encodes the
>>>>>>>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>>>>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>>>>>>>>> something we should be trying to define here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just catalog
>>>>>>>>>>>>>> naming consistency. The behavior around default-catalog itself introduces
>>>>>>>>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * When default-catalog is set to null, tables inside views
>>>>>>>>>>>>>> remain reachable after a catalog rename. But if it is set to a non-null
>>>>>>>>>>>>>> value, table references will break.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * default-catalog causes table references inside views to be
>>>>>>>>>>>>>> early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the
>>>>>>>>>>>>>> spec. There is no way to fully align Spark's behavior without making
>>>>>>>>>>>>>> invasive changes to the Spark SQL grammar and the View DataSource API
>>>>>>>>>>>>>> (specifically on the CREATE side). This challenge would extend to other
>>>>>>>>>>>>>> engines too. Both Spark and Trino set this field based on a heuristic in
>>>>>>>>>>>>>> today's implementation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with
>>>>>>>>>>>>>> that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities would
>>>>>>>>>>>>>> align around having table resolution behave inconsistently between queries
>>>>>>>>>>>>>> and views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog'
>>>>>>>>>>>>>>> was introduced, but I think we need to step back and align on what we are
>>>>>>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The issues we're discussing largely stem from using multiple
>>>>>>>>>>>>>>> engines with cross catalog references and configurations where catalog
>>>>>>>>>>>>>>> names are not aligned.  If we have multiple engines that all have the same
>>>>>>>>>>>>>>> catalog names/configurations, the current spec implementation is well
>>>>>>>>>>>>>>> defined for table resolution even across catalogs.  The 'default-catalog'
>>>>>>>>>>>>>>> (and namespace equivalent) was intended to address the resolution within
>>>>>>>>>>>>>>> the context of the sql text, not to address catalog/naming inconsistencies.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent to
>>>>>>>>>>>>>>> address the catalog naming/configuration and would argue that we shouldn't
>>>>>>>>>>>>>>> attempt to do that as part of the spec.  Inconsistently named catalogs are
>>>>>>>>>>>>>>> a reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We should support and advocate for consistency in catalog
>>>>>>>>>>>>>>> naming and define the spec along those lines.  The fact is that with all of
>>>>>>>>>>>>>>> the recent work that's gone into making catalogs pluggable, it makes more
>>>>>>>>>>>>>>> sense to just register catalog configuration with consistent names (even if
>>>>>>>>>>>>>>> you have to duplicate the configuration for supporting existing
>>>>>>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late
>>>>>>>>>>>>>>> binding resolution to swap in different catalogs or make references
>>>>>>>>>>>>>>> contextual, I feel like that is something we should strongly discourage as
>>>>>>>>>>>>>>> it leads to confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was that
>>>>>>>>>>>>>>>> every query engine uses it to store its session default catalog at the time
>>>>>>>>>>>>>>>> of creating the view. This way the view could be reused in another session.
>>>>>>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Generally we have different environments we want to support
>>>>>>>>>>>>>>>> with the view spec:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When the environment supports it, using consistent catalog
>>>>>>>>>>>>>>>> names can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>>>>>>>>> any issues.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This can be the case when different query engines refer to
>>>>>>>>>>>>>>>> the same physical catalog by different names. This often happens because
>>>>>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields from
>>>>>>>>>>>>>>>> the view spec completely.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level
>>>>>>>>>>>>>>>> default catalogs or namespaces.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as
>>>>>>>>>>>>>>>> they appear in the view SQL text.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it
>>>>>>>>>>>>>>>> should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the
>>>>>>>>>>>>>>>>> fallback creates. Also just to be clear, at this point and after clarifying
>>>>>>>>>>>>>>>>> the current spec intentions, I am convinced that we should remove the
>>>>>>>>>>>>>>>>> default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't
>>>>>>>>>>>>>>>>>> think we should design something that says first use this catalog, then
>>>>>>>>>>>>>>>>>> that catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no
>>>>>>>>>>>>>>>>>> value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>>> - Interoperability between different engine types is very
>>>>>>>>>>>>>>>>>> hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along
>>>>>>>>>>>>>>>>>> the lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier
>>>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the right approach
>>>>>>>>>>>>>>>>>> is really about templating.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a
>>>>>>>>>>>>>>>>>> view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I
>>>>>>>>>>>>>>>>>>> shared regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback should
>>>>>>>>>>>>>>>>>>>> be the catalog
>>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter
>>>>>>>>>>>>>>>>>>>> what the catalog
>>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog,
>>>>>>>>>>>>>>>>>>>> this is a
>>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog,
>>>>>>>>>>>>>>>>>>>> catalog names
>>>>>>>>>>>>>>>>>>>> should be included in the reference name already. So no
>>>>>>>>>>>>>>>>>>>> ambiguity
>>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage for
>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the
>>>>>>>>>>>>>>>>>>>> current fallback
>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for
>>>>>>>>>>>>>>>>>>>> not conforming
>>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to
>>>>>>>>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view
>>>>>>>>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist before its
>>>>>>>>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible in Spark’s
>>>>>>>>>>>>>>>>>>>> implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting
>>>>>>>>>>>>>>>>>>>> would require extending SQL syntax and engine APIs to promote it to a
>>>>>>>>>>>>>>>>>>>> first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a
>>>>>>>>>>>>>>>>>>>> creation-time default-catalog, it would create an inconsistent binding
>>>>>>>>>>>>>>>>>>>> model between tables vs views (early vs late), and between tables in views
>>>>>>>>>>>>>>>>>>>> and in queries (again early vs late). For example, views and tables in
>>>>>>>>>>>>>>>>>>>> queries can withstand default catalog renames, but tables cannot when they
>>>>>>>>>>>>>>>>>>>> are used inside views -- it even applies to views inside views, which makes
>>>>>>>>>>>>>>>>>>>> this very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the
>>>>>>>>>>>>>>>>>>>> "view catalog" where the view is going to be created in. It can use that
>>>>>>>>>>>>>>>>>>>> catalog to resolve partial table identifiers if "default-catalog" is not
>>>>>>>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where
>>>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT
>>>>>>>>>>>>>>>>>>>> * from sales.orders;
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA",
>>>>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same as just
>>>>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>> default to "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the tables
>>>>>>>>>>>>>>>>>>>> referenced in the view exist, how can we do that when default-catalog isn't
>>>>>>>>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet?
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do
>>>>>>>>>>>>>>>>>>>> we validate that a table exists without a default catalog, or do we always
>>>>>>>>>>>>>>>>>>>> use the current session catalog?
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding. Just
>>>>>>>>>>>>>>>>>>>> to clarify: when I referred to late binding as “similar” to the proposal, I
>>>>>>>>>>>>>>>>>>>> was acknowledging the distinction between view-level and table-level
>>>>>>>>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting
>>>>>>>>>>>>>>>>>>>> question and a potential gap: if default-catalog is only defined at query
>>>>>>>>>>>>>>>>>>>> time, how should resolution work during view creation? For example, if we
>>>>>>>>>>>>>>>>>>>> want to validate that the tables referenced in the view exist, how can we
>>>>>>>>>>>>>>>>>>>> do that when default-catalog isn't defined, since the view hasn't been
>>>>>>>>>>>>>>>>>>>> created or loaded yet?
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view
>>>>>>>>>>>>>>>>>>>> catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish
>>>>>>>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>>>>>>>>> still work.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late
>>>>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension to all partial
>>>>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. Resolving the view
>>>>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the view metadata in a
>>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced
>>>>>>>>>>>>>>>>>>>> in a SQL statement. That statement contains the information for the query
>>>>>>>>>>>>>>>>>>>> engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT *
>>>>>>>>>>>>>>>>>>>> FROM catalogB.namespace.view (after renaming catalogA to catalogB, and
>>>>>>>>>>>>>>>>>>>> keeping everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds
>>>>>>>>>>>>>>>>>>>> to me like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec:
>>>>>>>>>>>>>>>>>>>> The view is executed when it's being referenced in a SQL statement. That
>>>>>>>>>>>>>>>>>>>> statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because the
>>>>>>>>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For example
>>>>>>>>>>>>>>>>>>>> for the following statement:
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA",
>>>>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same as just
>>>>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view
>>>>>>>>>>>>>>>>>>>> spec, because it has the "closure" property. Because of the fact that the
>>>>>>>>>>>>>>>>>>>> "view catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point
>>>>>>>>>>>>>>>>>>>> before going deeper:
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback
>>>>>>>>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table identifiers alone,
>>>>>>>>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed fallback to the
>>>>>>>>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help
>>>>>>>>>>>>>>>>>>>> clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog”
>>>>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if default-catalog is
>>>>>>>>>>>>>>>>>>>> null?
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we can’t
>>>>>>>>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then defaulting to
>>>>>>>>>>>>>>>>>>>> it is problematic. And if views can't be created in the default catalog,
>>>>>>>>>>>>>>>>>>>> then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the
>>>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of "non-determinism".
>>>>>>>>>>>>>>>>>>>> You could be running the same query-engine and catalog-setup on different
>>>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is rather common), and
>>>>>>>>>>>>>>>>>>>> would be getting different results.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view
>>>>>>>>>>>>>>>>>>>> always produces the same results. The current behavior has some rough edges
>>>>>>>>>>>>>>>>>>>> in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the
>>>>>>>>>>>>>>>>>>>> setup more robust. Which is great. However, having UUID validation still
>>>>>>>>>>>>>>>>>>>> requires us to have a portable table identifier specification. Even if we
>>>>>>>>>>>>>>>>>>>> have the UUIDs of the referenced tables from the view, there simply isn't
>>>>>>>>>>>>>>>>>>>> an interface that let's us use those UUIDs. The catalog interface is
>>>>>>>>>>>>>>>>>>>> defined in terms of table identifiers.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and
>>>>>>>>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for
>>>>>>>>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving
>>>>>>>>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text with table
>>>>>>>>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use the catalog
>>>>>>>>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret or use.
>>>>>>>>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated catalog that
>>>>>>>>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In such
>>>>>>>>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in `my_catalog.namespace1.table1`)
>>>>>>>>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to underlying
>>>>>>>>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the catalog
>>>>>>>>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs actually behave
>>>>>>>>>>>>>>>>>>>> in practice.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have
>>>>>>>>>>>>>>>>>>>> some concerns with resolving catalog names on the fly with query engine
>>>>>>>>>>>>>>>>>>>> defined catalog names. This introduces some flexibility at first glance,
>>>>>>>>>>>>>>>>>>>> but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and
>>>>>>>>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a vote next
>>>>>>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog
>>>>>>>>>>>>>>>>>>>> names in the spec refer to logical catalogs—i.e., the first part of a
>>>>>>>>>>>>>>>>>>>> three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think
>>>>>>>>>>>>>>>>>>>> view portability is a very important topic for us to continue discussing as
>>>>>>>>>>>>>>>>>>>> it relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this
>>>>>>>>>>>>>>>>>>>> may impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem.
>>>>>>>>>>>>>>>>>>>> I've added a few comments to get a better understanding of how this will
>>>>>>>>>>>>>>>>>>>> look like in the actual implementation.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a
>>>>>>>>>>>>>>>>>>>> few community members to better understand how the current spec is being
>>>>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look
>>>>>>>>>>>>>>>>>>>> forward to your feedback and working together to move this forward so we
>>>>>>>>>>>>>>>>>>>> can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

Thanks Steven! So would you agree that resolution using default-catalog and
default-namespace does not provide full determinism, and requires a
supporting safety mechanism?

Thanks,
Walaa.

On Wed, May 7, 2025 at 10:30 PM Steven Wu <st...@gmail.com> wrote:

> > If the current model is considered deterministic, do you think
> `default-catalog` and `default-namespace` fields provide enough determinism
> to eliminate the need for UUIDs when storing table identifiers?
>
> I am fine with storing UUIDs for table identifiers in the view. Basically,
> view creation resolves all referenced tables/views with UUIDs. View
> consumers can validate resolved tables/views with the stored UUIDs and fail
> the query if mismatch.
>
> The UUID change doesn't really change the table identifier resolution rule
> though. It is more of a safety protection.
>
> On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> Hi Steven,
>>
>> Thanks for the reply.
>>
>> > I agree with Dan that we shouldn't solve catalog naming in the Iceberg
>> view spec.
>>
>> To clarify, I don't believe the proposal is trying to solve catalog
>> naming. What it’s doing is simply this:
>>
>> * Proposing that table names inside views resolve the same way as they do
>> elsewhere (e.g., queries).
>> * Adopting a model that is already widely used and supported in the
>> existing ecosystem, which allows for:
>>     -- Renaming catalog aliases
>>     -- Swapping catalog implementations behind consistent names
>>     -- Having different default catalog names across engines that still
>> point to the same underlying tables
>>
>> These are common patterns in production data lakes. Saying Iceberg views
>> cannot operate in those environments feels unrealistic. In practice, it
>> means the spec breaks down in situations that users encounter regularly.
>>
>> > The recommendation of using engines’ current catalog and database can
>> cause context-dependent resolution results.
>>
>> * As noted in the doc and earlier replies, fixing a catalog name doesn’t
>> actually guarantee determinism either. All the failure scenarios above
>> still apply even when a default-catalog is stored.
>> * The current spec also allows default-catalog to be null, in which case
>> it falls back to the view’s catalog, yet that catalog is determined based
>> on how the view is referenced in the query, which would be considered
>> non-deterministic based on the same criteria you shared.
>> * The only true form of determinism here is UUID-based validation, which
>> protects against silent drift in any resolution model.
>>
>> If the current model is considered deterministic, do you think
>> `default-catalog` and `default-namespace` fields provide enough determinism
>> to eliminate the need for UUIDs when storing table identifiers?
>> Or put another way: Would you be comfortable relying solely on
>> default-catalog + default-namespace + table name to re-identify the correct
>> table, without UUID validation?
>>
>> +1 on involving other communities. I’m happy to help facilitate a
>> cross-community discussion if we aren’t able to reach a resolution here.
>>
>> Thanks,
>> Walaa.
>>
>>
>>
>> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com> wrote:
>>
>>> I agree with Dan that we shouldn't solve catalog naming in the Iceberg
>>> view spec. I am not convinced that the proposed change will make the table
>>> identifier resolution more clear and portable. The recommendation of using
>>> engines' current catalog and database can cause context dependent
>>> resolution results, which seems non-deterministic to me.
>>>
>>> Walaa, you raised a point in the doc that the current catalog resolution
>>> logic (default-catalog field, then view catalog) is challenging and
>>> unrealistic for engines (like Spark and Trino). It will be great to get
>>> more inputs from the broader community on this part.
>>>
>>>
>>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:
>>>
>>>> In Spark, I believe that the USE commands sets the current catalog and
>>>> namespace.  This affects both where the view is created and how unqualified
>>>> table identifiers are resolved.  I also don't see an issue with saving the
>>>> current catalog and namespace into the view metadata's default-catalog and
>>>> default-namespace fields.
>>>>
>>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> > I think that's the lesser evil compared to Iceberg specifying how
>>>>> engines should resolve identifiers
>>>>>
>>>>> I think this is also similar to the previous point. It is the other
>>>>> way around. Right now the spec dictates how to resolve (through employing a
>>>>> view-specific `default-catalog` field). The proposal is suggesting to get
>>>>> out of this space and let engines handle it similar to how they handle all
>>>>> identifiers.
>>>>>
>>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>>
>>>>>> Benny, I think this is a misconception or miscommunication. The USE
>>>>>> command has no impact on the `default-catalog` field. In fact, the
>>>>>> proposal's direction is exactly to establish that USE command should
>>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>>> is not the case under the current spec.
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com> wrote:
>>>>>>
>>>>>>> > there is no SQL construct today to explicitly set default-catalog
>>>>>>>
>>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>>
>>>>>>> I generally agree with Dan about requiring consistent catalog
>>>>>>> names.  I think that's the lesser evil compared to Iceberg specifying how
>>>>>>> engines should resolve identifiers.  Another thing to consider is that
>>>>>>> identifier resolution can be very expensive at query validation time if
>>>>>>> identifiers need to be looked up from a bunch of places.  Hopefully, it
>>>>>>> should be possible to define a view in such a way that identifiers can be
>>>>>>> resolved on the first try.
>>>>>>>
>>>>>>> Benny
>>>>>>>
>>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Rishabh,
>>>>>>>>
>>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>>> materialized views.
>>>>>>>>
>>>>>>>> We may be able to converge on this storage format even if we
>>>>>>>> haven't yet converged on the resolution fallback rules. I believe both
>>>>>>>> resolution strategies currently being discussed would still lead to storing
>>>>>>>> identifiers in this way.
>>>>>>>>
>>>>>>>> I'm supportive of moving forward with consensus on the identifier
>>>>>>>> storage format. That said, we may continue to run into questions related to
>>>>>>>> resolution during implementation. For example: Should the storage table
>>>>>>>> identifier follow the same default-catalog and default-namespace resolution
>>>>>>>> behavior as other table references?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Walaa.
>>>>>>>>
>>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello Walaa,
>>>>>>>>>
>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>
>>>>>>>>> I think we should decouple at least the MV Spec from the proposal
>>>>>>>>> to change the current behavior of view resolution.
>>>>>>>>>
>>>>>>>>> We can continue having the discussion if the current view spec
>>>>>>>>> needs to be changed or not. Based on the decision at a later point if
>>>>>>>>> required we can update the view resolution rule.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Rishabh
>>>>>>>>>
>>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Correction of typo: both engines seem to set default-catalog to
>>>>>>>>>> the view catalog if it is defined, or to null if the view catalog is not
>>>>>>>>>> defined.
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>
>>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>>
>>>>>>>>>>> I agree that catalog renaming is an environmental event, but
>>>>>>>>>>> it's a real one that happens frequently in practice.
>>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something as
>>>>>>>>>>> common as catalog renaming feels very restrictive, and could make the spec
>>>>>>>>>>> less practical, even unusable, for real-world deployments.
>>>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>>>
>>>>>>>>>>> Further, it's worth noting that the table spec is resilient to
>>>>>>>>>>> catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>>>>> Both specifications are deterministic in their definition, but
>>>>>>>>>>> one is more fragile to environmental changes than the other. Improving
>>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>>>> more portable over time.
>>>>>>>>>>>
>>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>>> Today, both engines seem to set default-catalog to null if the
>>>>>>>>>>> view catalog is defined, or to the view catalog if not.
>>>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Walaa
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dw...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Walaa,
>>>>>>>>>>>>
>>>>>>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>>>>>>
>>>>>>>>>>>> This problem stems from the exact environmental/configuration
>>>>>>>>>>>> issue that we should not be trying to address.  I don't think we would
>>>>>>>>>>>> expect references to survive a catalog rename.  That's not something
>>>>>>>>>>>> covered by the spec and needs to be handled separately as a platform-level
>>>>>>>>>>>> migration specific to the affected environment.
>>>>>>>>>>>>
>>>>>>>>>>>> The identifier resolution logic is clear and deterministic.  It
>>>>>>>>>>>> should not matter whether an engine resolves and encodes the
>>>>>>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>>
>>>>>>>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>>>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>>>>>>>> something we should be trying to define here.
>>>>>>>>>>>>
>>>>>>>>>>>> -Dan
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just catalog
>>>>>>>>>>>>> naming consistency. The behavior around default-catalog itself introduces
>>>>>>>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * When default-catalog is set to null, tables inside views
>>>>>>>>>>>>> remain reachable after a catalog rename. But if it is set to a non-null
>>>>>>>>>>>>> value, table references will break.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * default-catalog causes table references inside views to be
>>>>>>>>>>>>> early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the spec.
>>>>>>>>>>>>> There is no way to fully align Spark's behavior without making invasive
>>>>>>>>>>>>> changes to the Spark SQL grammar and the View DataSource API (specifically
>>>>>>>>>>>>> on the CREATE side). This challenge would extend to other engines too. Both
>>>>>>>>>>>>> Spark and Trino set this field based on a heuristic in today's
>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with
>>>>>>>>>>>>> that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>>
>>>>>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>>>>>> Also, do you believe the Spark and Trino communities would
>>>>>>>>>>>>> align around having table resolution behave inconsistently between queries
>>>>>>>>>>>>> and views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Walaa
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was
>>>>>>>>>>>>>> introduced, but I think we need to step back and align on what we are
>>>>>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The issues we're discussing largely stem from using multiple
>>>>>>>>>>>>>> engines with cross catalog references and configurations where catalog
>>>>>>>>>>>>>> names are not aligned.  If we have multiple engines that all have the same
>>>>>>>>>>>>>> catalog names/configurations, the current spec implementation is well
>>>>>>>>>>>>>> defined for table resolution even across catalogs.  The 'default-catalog'
>>>>>>>>>>>>>> (and namespace equivalent) was intended to address the resolution within
>>>>>>>>>>>>>> the context of the sql text, not to address catalog/naming inconsistencies.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I feel like we're trying to adapt the original intent to
>>>>>>>>>>>>>> address the catalog naming/configuration and would argue that we shouldn't
>>>>>>>>>>>>>> attempt to do that as part of the spec.  Inconsistently named catalogs are
>>>>>>>>>>>>>> a reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We should support and advocate for consistency in catalog
>>>>>>>>>>>>>> naming and define the spec along those lines.  The fact is that with all of
>>>>>>>>>>>>>> the recent work that's gone into making catalogs pluggable, it makes more
>>>>>>>>>>>>>> sense to just register catalog configuration with consistent names (even if
>>>>>>>>>>>>>> you have to duplicate the configuration for supporting existing
>>>>>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If the goal is to create clever ways to hack the late binding
>>>>>>>>>>>>>> resolution to swap in different catalogs or make references contextual, I
>>>>>>>>>>>>>> feel like that is something we should strongly discourage as it leads to
>>>>>>>>>>>>>> confusion about what is resolved as part of the query.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think the intention with the "default-catalog" was that
>>>>>>>>>>>>>>> every query engine uses it to store its session default catalog at the time
>>>>>>>>>>>>>>> of creating the view. This way the view could be reused in another session.
>>>>>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Generally we have different environments we want to support
>>>>>>>>>>>>>>> with the view spec:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When the environment supports it, using consistent catalog
>>>>>>>>>>>>>>> names can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>>>>>>>> any issues.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This can be the case when different query engines refer to
>>>>>>>>>>>>>>> the same physical catalog by different names. This often happens because
>>>>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields from
>>>>>>>>>>>>>>> the view spec completely.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level
>>>>>>>>>>>>>>> default catalogs or namespaces.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as
>>>>>>>>>>>>>>> they appear in the view SQL text.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it
>>>>>>>>>>>>>>> should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the
>>>>>>>>>>>>>>>> fallback creates. Also just to be clear, at this point and after clarifying
>>>>>>>>>>>>>>>> the current spec intentions, I am convinced that we should remove the
>>>>>>>>>>>>>>>> default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <
>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't
>>>>>>>>>>>>>>>>> think we should design something that says first use this catalog, then
>>>>>>>>>>>>>>>>> that catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no
>>>>>>>>>>>>>>>>> value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>>> - Interoperability between different engine types is very
>>>>>>>>>>>>>>>>> hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the
>>>>>>>>>>>>>>>>> lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier
>>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the right approach
>>>>>>>>>>>>>>>>> is really about templating.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a
>>>>>>>>>>>>>>>>> view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I
>>>>>>>>>>>>>>>>>> shared regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback should
>>>>>>>>>>>>>>>>>>> be the catalog
>>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter what
>>>>>>>>>>>>>>>>>>> the catalog
>>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog,
>>>>>>>>>>>>>>>>>>> this is a
>>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog,
>>>>>>>>>>>>>>>>>>> catalog names
>>>>>>>>>>>>>>>>>>> should be included in the reference name already. So no
>>>>>>>>>>>>>>>>>>> ambiguity
>>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage for
>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the
>>>>>>>>>>>>>>>>>>> current fallback
>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for
>>>>>>>>>>>>>>>>>>> not conforming
>>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to
>>>>>>>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view
>>>>>>>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist before its
>>>>>>>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible in Spark’s
>>>>>>>>>>>>>>>>>>> implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting
>>>>>>>>>>>>>>>>>>> would require extending SQL syntax and engine APIs to promote it to a
>>>>>>>>>>>>>>>>>>> first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time
>>>>>>>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding model between
>>>>>>>>>>>>>>>>>>> tables vs views (early vs late), and between tables in views and in queries
>>>>>>>>>>>>>>>>>>> (again early vs late). For example, views and tables in queries can
>>>>>>>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when they are used
>>>>>>>>>>>>>>>>>>> inside views -- it even applies to views inside views, which makes this
>>>>>>>>>>>>>>>>>>> very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the
>>>>>>>>>>>>>>>>>>> "view catalog" where the view is going to be created in. It can use that
>>>>>>>>>>>>>>>>>>> catalog to resolve partial table identifiers if "default-catalog" is not
>>>>>>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where
>>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>>>>>>> "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the tables
>>>>>>>>>>>>>>>>>>> referenced in the view exist, how can we do that when default-catalog isn't
>>>>>>>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet?
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we
>>>>>>>>>>>>>>>>>>> validate that a table exists without a default catalog, or do we always use
>>>>>>>>>>>>>>>>>>> the current session catalog?
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding. Just
>>>>>>>>>>>>>>>>>>> to clarify: when I referred to late binding as “similar” to the proposal, I
>>>>>>>>>>>>>>>>>>> was acknowledging the distinction between view-level and table-level
>>>>>>>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting question
>>>>>>>>>>>>>>>>>>> and a potential gap: if default-catalog is only defined at query time, how
>>>>>>>>>>>>>>>>>>> should resolution work during view creation? For example, if we want to
>>>>>>>>>>>>>>>>>>> validate that the tables referenced in the view exist, how can we do that
>>>>>>>>>>>>>>>>>>> when default-catalog isn't defined, since the view hasn't been created or
>>>>>>>>>>>>>>>>>>> loaded yet?
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view
>>>>>>>>>>>>>>>>>>> catalog is resolved at query time.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish
>>>>>>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>>>>>>>> still work.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late
>>>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension to all partial
>>>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. Resolving the view
>>>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the view metadata in a
>>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced
>>>>>>>>>>>>>>>>>>> in a SQL statement. That statement contains the information for the query
>>>>>>>>>>>>>>>>>>> engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT *
>>>>>>>>>>>>>>>>>>> FROM catalogB.namespace.view (after renaming catalogA to catalogB, and
>>>>>>>>>>>>>>>>>>> keeping everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to
>>>>>>>>>>>>>>>>>>> me like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec:
>>>>>>>>>>>>>>>>>>> The view is executed when it's being referenced in a SQL statement. That
>>>>>>>>>>>>>>>>>>> statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because the
>>>>>>>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For example
>>>>>>>>>>>>>>>>>>> for the following statement:
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS
>>>>>>>>>>>>>>>>>>> SELECT * from sales.orders;
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA",
>>>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same as just
>>>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view
>>>>>>>>>>>>>>>>>>> spec, because it has the "closure" property. Because of the fact that the
>>>>>>>>>>>>>>>>>>> "view catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point
>>>>>>>>>>>>>>>>>>> before going deeper:
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback
>>>>>>>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table identifiers alone,
>>>>>>>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed fallback to the
>>>>>>>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help
>>>>>>>>>>>>>>>>>>> clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog”
>>>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if default-catalog is
>>>>>>>>>>>>>>>>>>> null?
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we can’t
>>>>>>>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then defaulting to
>>>>>>>>>>>>>>>>>>> it is problematic. And if views can't be created in the default catalog,
>>>>>>>>>>>>>>>>>>> then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the
>>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of "non-determinism".
>>>>>>>>>>>>>>>>>>> You could be running the same query-engine and catalog-setup on different
>>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is rather common), and
>>>>>>>>>>>>>>>>>>> would be getting different results.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view
>>>>>>>>>>>>>>>>>>> always produces the same results. The current behavior has some rough edges
>>>>>>>>>>>>>>>>>>> in very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup
>>>>>>>>>>>>>>>>>>> more robust. Which is great. However, having UUID validation still requires
>>>>>>>>>>>>>>>>>>> us to have a portable table identifier specification. Even if we have the
>>>>>>>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply isn't an
>>>>>>>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog interface is defined
>>>>>>>>>>>>>>>>>>> in terms of table identifiers.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and
>>>>>>>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for
>>>>>>>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving
>>>>>>>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text with table
>>>>>>>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use the catalog
>>>>>>>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret or use.
>>>>>>>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated catalog that
>>>>>>>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In such
>>>>>>>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in `my_catalog.namespace1.table1`)
>>>>>>>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to underlying
>>>>>>>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the catalog
>>>>>>>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs actually behave
>>>>>>>>>>>>>>>>>>> in practice.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have
>>>>>>>>>>>>>>>>>>> some concerns with resolving catalog names on the fly with query engine
>>>>>>>>>>>>>>>>>>> defined catalog names. This introduces some flexibility at first glance,
>>>>>>>>>>>>>>>>>>> but also makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and
>>>>>>>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a vote next
>>>>>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog
>>>>>>>>>>>>>>>>>>> names in the spec refer to logical catalogs—i.e., the first part of a
>>>>>>>>>>>>>>>>>>> three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think
>>>>>>>>>>>>>>>>>>> view portability is a very important topic for us to continue discussing as
>>>>>>>>>>>>>>>>>>> it relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may
>>>>>>>>>>>>>>>>>>> impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've
>>>>>>>>>>>>>>>>>>> added a few comments to get a better understanding of how this will look
>>>>>>>>>>>>>>>>>>> like in the actual implementation.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a
>>>>>>>>>>>>>>>>>>> few community members to better understand how the current spec is being
>>>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look
>>>>>>>>>>>>>>>>>>> forward to your feedback and working together to move this forward so we
>>>>>>>>>>>>>>>>>>> can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Steven Wu <st...@gmail.com>]

> If the current model is considered deterministic, do you think
`default-catalog` and `default-namespace` fields provide enough determinism
to eliminate the need for UUIDs when storing table identifiers?

I am fine with storing UUIDs for table identifiers in the view. Basically,
view creation resolves all referenced tables/views with UUIDs. View
consumers can validate resolved tables/views with the stored UUIDs and fail
the query if mismatch.

The UUID change doesn't really change the table identifier resolution rule
though. It is more of a safety protection.

On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> Hi Steven,
>
> Thanks for the reply.
>
> > I agree with Dan that we shouldn't solve catalog naming in the Iceberg
> view spec.
>
> To clarify, I don't believe the proposal is trying to solve catalog
> naming. What it’s doing is simply this:
>
> * Proposing that table names inside views resolve the same way as they do
> elsewhere (e.g., queries).
> * Adopting a model that is already widely used and supported in the
> existing ecosystem, which allows for:
>     -- Renaming catalog aliases
>     -- Swapping catalog implementations behind consistent names
>     -- Having different default catalog names across engines that still
> point to the same underlying tables
>
> These are common patterns in production data lakes. Saying Iceberg views
> cannot operate in those environments feels unrealistic. In practice, it
> means the spec breaks down in situations that users encounter regularly.
>
> > The recommendation of using engines’ current catalog and database can
> cause context-dependent resolution results.
>
> * As noted in the doc and earlier replies, fixing a catalog name doesn’t
> actually guarantee determinism either. All the failure scenarios above
> still apply even when a default-catalog is stored.
> * The current spec also allows default-catalog to be null, in which case
> it falls back to the view’s catalog, yet that catalog is determined based
> on how the view is referenced in the query, which would be considered
> non-deterministic based on the same criteria you shared.
> * The only true form of determinism here is UUID-based validation, which
> protects against silent drift in any resolution model.
>
> If the current model is considered deterministic, do you think
> `default-catalog` and `default-namespace` fields provide enough determinism
> to eliminate the need for UUIDs when storing table identifiers?
> Or put another way: Would you be comfortable relying solely on
> default-catalog + default-namespace + table name to re-identify the correct
> table, without UUID validation?
>
> +1 on involving other communities. I’m happy to help facilitate a
> cross-community discussion if we aren’t able to reach a resolution here.
>
> Thanks,
> Walaa.
>
>
>
> On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com> wrote:
>
>> I agree with Dan that we shouldn't solve catalog naming in the Iceberg
>> view spec. I am not convinced that the proposed change will make the table
>> identifier resolution more clear and portable. The recommendation of using
>> engines' current catalog and database can cause context dependent
>> resolution results, which seems non-deterministic to me.
>>
>> Walaa, you raised a point in the doc that the current catalog resolution
>> logic (default-catalog field, then view catalog) is challenging and
>> unrealistic for engines (like Spark and Trino). It will be great to get
>> more inputs from the broader community on this part.
>>
>>
>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:
>>
>>> In Spark, I believe that the USE commands sets the current catalog and
>>> namespace.  This affects both where the view is created and how unqualified
>>> table identifiers are resolved.  I also don't see an issue with saving the
>>> current catalog and namespace into the view metadata's default-catalog and
>>> default-namespace fields.
>>>
>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> > I think that's the lesser evil compared to Iceberg specifying how
>>>> engines should resolve identifiers
>>>>
>>>> I think this is also similar to the previous point. It is the other way
>>>> around. Right now the spec dictates how to resolve (through employing a
>>>> view-specific `default-catalog` field). The proposal is suggesting to get
>>>> out of this space and let engines handle it similar to how they handle all
>>>> identifiers.
>>>>
>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> > I thought "default-catalog" could be set via the USE command.
>>>>>
>>>>> Benny, I think this is a misconception or miscommunication. The USE
>>>>> command has no impact on the `default-catalog` field. In fact, the
>>>>> proposal's direction is exactly to establish that USE command should
>>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>>> is not the case under the current spec.
>>>>>
>>>>>
>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com> wrote:
>>>>>
>>>>>> > there is no SQL construct today to explicitly set default-catalog
>>>>>>
>>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>>
>>>>>> I generally agree with Dan about requiring consistent catalog names.
>>>>>> I think that's the lesser evil compared to Iceberg specifying how engines
>>>>>> should resolve identifiers.  Another thing to consider is that identifier
>>>>>> resolution can be very expensive at query validation time if identifiers
>>>>>> need to be looked up from a bunch of places.  Hopefully, it should be
>>>>>> possible to define a view in such a way that identifiers can be resolved on
>>>>>> the first try.
>>>>>>
>>>>>> Benny
>>>>>>
>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Rishabh,
>>>>>>>
>>>>>>> You're right that the proposal touches on two aspects, and
>>>>>>> resolution rules are one of them. The other aspect is the proposal's
>>>>>>> position that table identifiers should be stored in metadata exactly as
>>>>>>> they appear in the view text (e.g., even if they're two-part or partially
>>>>>>> qualified), along with their corresponding UUIDs for validation. This
>>>>>>> applies both to referenced input tables and the storage table identifier in
>>>>>>> materialized views.
>>>>>>>
>>>>>>> We may be able to converge on this storage format even if we haven't
>>>>>>> yet converged on the resolution fallback rules. I believe both resolution
>>>>>>> strategies currently being discussed would still lead to storing
>>>>>>> identifiers in this way.
>>>>>>>
>>>>>>> I'm supportive of moving forward with consensus on the identifier
>>>>>>> storage format. That said, we may continue to run into questions related to
>>>>>>> resolution during implementation. For example: Should the storage table
>>>>>>> identifier follow the same default-catalog and default-namespace resolution
>>>>>>> behavior as other table references?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Walaa.
>>>>>>>
>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello Walaa,
>>>>>>>>
>>>>>>>> Thanks for starting this discussion.
>>>>>>>>
>>>>>>>> I think we should decouple at least the MV Spec from the proposal
>>>>>>>> to change the current behavior of view resolution.
>>>>>>>>
>>>>>>>> We can continue having the discussion if the current view spec
>>>>>>>> needs to be changed or not. Based on the decision at a later point if
>>>>>>>> required we can update the view resolution rule.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Rishabh
>>>>>>>>
>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Correction of typo: both engines seem to set default-catalog to
>>>>>>>>> the view catalog if it is defined, or to null if the view catalog is not
>>>>>>>>> defined.
>>>>>>>>>
>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Dan,
>>>>>>>>>>
>>>>>>>>>> Thanks again for your response.
>>>>>>>>>>
>>>>>>>>>> I agree that catalog renaming is an environmental event, but it's
>>>>>>>>>> a real one that happens frequently in practice.
>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something as
>>>>>>>>>> common as catalog renaming feels very restrictive, and could make the spec
>>>>>>>>>> less practical, even unusable, for real-world deployments.
>>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>>
>>>>>>>>>> Further, it's worth noting that the table spec is resilient to
>>>>>>>>>> catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>>>> Both specifications are deterministic in their definition, but
>>>>>>>>>> one is more fragile to environmental changes than the other. Improving
>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>>> more portable over time.
>>>>>>>>>>
>>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>>> Today, both engines seem to set default-catalog to null if the
>>>>>>>>>> view catalog is defined, or to the view catalog if not.
>>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Walaa
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dw...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Walaa,
>>>>>>>>>>>
>>>>>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>>>>>
>>>>>>>>>>> This problem stems from the exact environmental/configuration
>>>>>>>>>>> issue that we should not be trying to address.  I don't think we would
>>>>>>>>>>> expect references to survive a catalog rename.  That's not something
>>>>>>>>>>> covered by the spec and needs to be handled separately as a platform-level
>>>>>>>>>>> migration specific to the affected environment.
>>>>>>>>>>>
>>>>>>>>>>> The identifier resolution logic is clear and deterministic.  It
>>>>>>>>>>> should not matter whether an engine resolves and encodes the
>>>>>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>>>>>
>>>>>>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>>>>>>> something we should be trying to define here.
>>>>>>>>>>>
>>>>>>>>>>> -Dan
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>>
>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just catalog
>>>>>>>>>>>> naming consistency. The behavior around default-catalog itself introduces
>>>>>>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>>>>>>> For example:
>>>>>>>>>>>>
>>>>>>>>>>>> * When default-catalog is set to null, tables inside views
>>>>>>>>>>>> remain reachable after a catalog rename. But if it is set to a non-null
>>>>>>>>>>>> value, table references will break.
>>>>>>>>>>>>
>>>>>>>>>>>> * default-catalog causes table references inside views to be
>>>>>>>>>>>> early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>>>
>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the spec.
>>>>>>>>>>>> There is no way to fully align Spark's behavior without making invasive
>>>>>>>>>>>> changes to the Spark SQL grammar and the View DataSource API (specifically
>>>>>>>>>>>> on the CREATE side). This challenge would extend to other engines too. Both
>>>>>>>>>>>> Spark and Trino set this field based on a heuristic in today's
>>>>>>>>>>>> implementation.
>>>>>>>>>>>>
>>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>>
>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with
>>>>>>>>>>>> that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>>
>>>>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>>>>> Also, do you believe the Spark and Trino communities would
>>>>>>>>>>>> align around having table resolution behave inconsistently between queries
>>>>>>>>>>>> and views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Walaa
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <
>>>>>>>>>>>> dweeks@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was
>>>>>>>>>>>>> introduced, but I think we need to step back and align on what we are
>>>>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The issues we're discussing largely stem from using multiple
>>>>>>>>>>>>> engines with cross catalog references and configurations where catalog
>>>>>>>>>>>>> names are not aligned.  If we have multiple engines that all have the same
>>>>>>>>>>>>> catalog names/configurations, the current spec implementation is well
>>>>>>>>>>>>> defined for table resolution even across catalogs.  The 'default-catalog'
>>>>>>>>>>>>> (and namespace equivalent) was intended to address the resolution within
>>>>>>>>>>>>> the context of the sql text, not to address catalog/naming inconsistencies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I feel like we're trying to adapt the original intent to
>>>>>>>>>>>>> address the catalog naming/configuration and would argue that we shouldn't
>>>>>>>>>>>>> attempt to do that as part of the spec.  Inconsistently named catalogs are
>>>>>>>>>>>>> a reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We should support and advocate for consistency in catalog
>>>>>>>>>>>>> naming and define the spec along those lines.  The fact is that with all of
>>>>>>>>>>>>> the recent work that's gone into making catalogs pluggable, it makes more
>>>>>>>>>>>>> sense to just register catalog configuration with consistent names (even if
>>>>>>>>>>>>> you have to duplicate the configuration for supporting existing
>>>>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If the goal is to create clever ways to hack the late binding
>>>>>>>>>>>>> resolution to swap in different catalogs or make references contextual, I
>>>>>>>>>>>>> feel like that is something we should strongly discourage as it leads to
>>>>>>>>>>>>> confusion about what is resolved as part of the query.
>>>>>>>>>>>>>
>>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think the intention with the "default-catalog" was that
>>>>>>>>>>>>>> every query engine uses it to store its session default catalog at the time
>>>>>>>>>>>>>> of creating the view. This way the view could be reused in another session.
>>>>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Generally we have different environments we want to support
>>>>>>>>>>>>>> with the view spec:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When the environment supports it, using consistent catalog
>>>>>>>>>>>>>> names can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>>>>>>> any issues.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This can be the case when different query engines refer to
>>>>>>>>>>>>>> the same physical catalog by different names. This often happens because
>>>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields from
>>>>>>>>>>>>>> the view spec completely.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level
>>>>>>>>>>>>>> default catalogs or namespaces.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as they
>>>>>>>>>>>>>> appear in the view SQL text.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it
>>>>>>>>>>>>>> should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the
>>>>>>>>>>>>>>> fallback creates. Also just to be clear, at this point and after clarifying
>>>>>>>>>>>>>>> the current spec intentions, I am convinced that we should remove the
>>>>>>>>>>>>>>> default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of
>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not set".
>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and
>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise SQL.
>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't
>>>>>>>>>>>>>>>> think we should design something that says first use this catalog, then
>>>>>>>>>>>>>>>> that catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no
>>>>>>>>>>>>>>>> value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>>> - Interoperability between different engine types is very
>>>>>>>>>>>>>>>> hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the
>>>>>>>>>>>>>>>> lines of something Russell proposed last year:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier problem
>>>>>>>>>>>>>>>> through the lens of catalog resolution but maybe the right approach is
>>>>>>>>>>>>>>>> really about templating.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a
>>>>>>>>>>>>>>>> view to span catalogs to support non-Iceberg tables.   Also, the
>>>>>>>>>>>>>>>> default-catalog property could be templated as well.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I
>>>>>>>>>>>>>>>>> shared regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback should
>>>>>>>>>>>>>>>>>> be the catalog
>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter what
>>>>>>>>>>>>>>>>>> the catalog
>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog,
>>>>>>>>>>>>>>>>>> this is a
>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog,
>>>>>>>>>>>>>>>>>> catalog names
>>>>>>>>>>>>>>>>>> should be included in the reference name already. So no
>>>>>>>>>>>>>>>>>> ambiguity
>>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage for
>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the
>>>>>>>>>>>>>>>>>> current fallback
>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for
>>>>>>>>>>>>>>>>>> not conforming
>>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to
>>>>>>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view
>>>>>>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist before its
>>>>>>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible in Spark’s
>>>>>>>>>>>>>>>>>> implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting
>>>>>>>>>>>>>>>>>> would require extending SQL syntax and engine APIs to promote it to a
>>>>>>>>>>>>>>>>>> first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time
>>>>>>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding model between
>>>>>>>>>>>>>>>>>> tables vs views (early vs late), and between tables in views and in queries
>>>>>>>>>>>>>>>>>> (again early vs late). For example, views and tables in queries can
>>>>>>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when they are used
>>>>>>>>>>>>>>>>>> inside views -- it even applies to views inside views, which makes this
>>>>>>>>>>>>>>>>>> very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW
>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is being created
>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve the view
>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time.
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the "view
>>>>>>>>>>>>>>>>>> catalog" where the view is going to be created in. It can use that catalog
>>>>>>>>>>>>>>>>>> to resolve partial table identifiers if "default-catalog" is not set.
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where
>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a different catalog
>>>>>>>>>>>>>>>>>> compared to using them outside of a view.
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>>>>>> "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the tables
>>>>>>>>>>>>>>>>>> referenced in the view exist, how can we do that when default-catalog isn't
>>>>>>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet?
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we
>>>>>>>>>>>>>>>>>> validate that a table exists without a default catalog, or do we always use
>>>>>>>>>>>>>>>>>> the current session catalog?
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding. Just
>>>>>>>>>>>>>>>>>> to clarify: when I referred to late binding as “similar” to the proposal, I
>>>>>>>>>>>>>>>>>> was acknowledging the distinction between view-level and table-level
>>>>>>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting question
>>>>>>>>>>>>>>>>>> and a potential gap: if default-catalog is only defined at query time, how
>>>>>>>>>>>>>>>>>> should resolution work during view creation? For example, if we want to
>>>>>>>>>>>>>>>>>> validate that the tables referenced in the view exist, how can we do that
>>>>>>>>>>>>>>>>>> when default-catalog isn't defined, since the view hasn't been created or
>>>>>>>>>>>>>>>>>> loaded yet?
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view catalog
>>>>>>>>>>>>>>>>>> is resolved at query time.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish
>>>>>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>>>>>>> still work.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late
>>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension to all partial
>>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. Resolving the view
>>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the view metadata in a
>>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced in
>>>>>>>>>>>>>>>>>> a SQL statement. That statement contains the information for the query
>>>>>>>>>>>>>>>>>> engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT * FROM
>>>>>>>>>>>>>>>>>> catalogB.namespace.view (after renaming catalogA to catalogB, and keeping
>>>>>>>>>>>>>>>>>> everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to
>>>>>>>>>>>>>>>>>> me like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec:
>>>>>>>>>>>>>>>>>> The view is executed when it's being referenced in a SQL statement. That
>>>>>>>>>>>>>>>>>> statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because the
>>>>>>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For example
>>>>>>>>>>>>>>>>>> for the following statement:
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS SELECT
>>>>>>>>>>>>>>>>>> * from sales.orders;
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA",
>>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same as just
>>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view
>>>>>>>>>>>>>>>>>> spec, because it has the "closure" property. Because of the fact that the
>>>>>>>>>>>>>>>>>> "view catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point
>>>>>>>>>>>>>>>>>> before going deeper:
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback
>>>>>>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table identifiers alone,
>>>>>>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed fallback to the
>>>>>>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help
>>>>>>>>>>>>>>>>>> clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog”
>>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if default-catalog is
>>>>>>>>>>>>>>>>>> null?
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we can’t
>>>>>>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then defaulting to
>>>>>>>>>>>>>>>>>> it is problematic. And if views can't be created in the default catalog,
>>>>>>>>>>>>>>>>>> then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the current
>>>>>>>>>>>>>>>>>> session setup is in my opinion the definition of "non-determinism". You
>>>>>>>>>>>>>>>>>> could be running the same query-engine and catalog-setup on different days,
>>>>>>>>>>>>>>>>>> with different default session catalogs (which is rather common), and would
>>>>>>>>>>>>>>>>>> be getting different results.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view always
>>>>>>>>>>>>>>>>>> produces the same results. The current behavior has some rough edges in
>>>>>>>>>>>>>>>>>> very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup
>>>>>>>>>>>>>>>>>> more robust. Which is great. However, having UUID validation still requires
>>>>>>>>>>>>>>>>>> us to have a portable table identifier specification. Even if we have the
>>>>>>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply isn't an
>>>>>>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog interface is defined
>>>>>>>>>>>>>>>>>> in terms of table identifiers.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and
>>>>>>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for
>>>>>>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving
>>>>>>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text with table
>>>>>>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use the catalog
>>>>>>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret or use.
>>>>>>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated catalog that
>>>>>>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In such
>>>>>>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in `my_catalog.namespace1.table1`)
>>>>>>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to underlying
>>>>>>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the catalog
>>>>>>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs actually behave
>>>>>>>>>>>>>>>>>> in practice.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have some
>>>>>>>>>>>>>>>>>> concerns with resolving catalog names on the fly with query engine defined
>>>>>>>>>>>>>>>>>> catalog names. This introduces some flexibility at first glance, but also
>>>>>>>>>>>>>>>>>> makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and
>>>>>>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a vote next
>>>>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog
>>>>>>>>>>>>>>>>>> names in the spec refer to logical catalogs—i.e., the first part of a
>>>>>>>>>>>>>>>>>> three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think view
>>>>>>>>>>>>>>>>>> portability is a very important topic for us to continue discussing as it
>>>>>>>>>>>>>>>>>> relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may
>>>>>>>>>>>>>>>>>> impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've
>>>>>>>>>>>>>>>>>> added a few comments to get a better understanding of how this will look
>>>>>>>>>>>>>>>>>> like in the actual implementation.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin
>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our
>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from Iceberg metadata, a
>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in relation to the MV
>>>>>>>>>>>>>>>>>> (materialized view) extensions.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a few
>>>>>>>>>>>>>>>>>> community members to better understand how the current spec is being
>>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look
>>>>>>>>>>>>>>>>>> forward to your feedback and working together to move this forward so we
>>>>>>>>>>>>>>>>>> can finalize the MV spec as well.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

Hi Steven,

Thanks for the reply.

> I agree with Dan that we shouldn't solve catalog naming in the Iceberg
view spec.

To clarify, I don't believe the proposal is trying to solve catalog naming.
What it’s doing is simply this:

* Proposing that table names inside views resolve the same way as they do
elsewhere (e.g., queries).
* Adopting a model that is already widely used and supported in the
existing ecosystem, which allows for:
    -- Renaming catalog aliases
    -- Swapping catalog implementations behind consistent names
    -- Having different default catalog names across engines that still
point to the same underlying tables

These are common patterns in production data lakes. Saying Iceberg views
cannot operate in those environments feels unrealistic. In practice, it
means the spec breaks down in situations that users encounter regularly.

> The recommendation of using engines’ current catalog and database can
cause context-dependent resolution results.

* As noted in the doc and earlier replies, fixing a catalog name doesn’t
actually guarantee determinism either. All the failure scenarios above
still apply even when a default-catalog is stored.
* The current spec also allows default-catalog to be null, in which case it
falls back to the view’s catalog, yet that catalog is determined based on
how the view is referenced in the query, which would be considered
non-deterministic based on the same criteria you shared.
* The only true form of determinism here is UUID-based validation, which
protects against silent drift in any resolution model.

If the current model is considered deterministic, do you think
`default-catalog` and `default-namespace` fields provide enough determinism
to eliminate the need for UUIDs when storing table identifiers?
Or put another way: Would you be comfortable relying solely on
default-catalog + default-namespace + table name to re-identify the correct
table, without UUID validation?

+1 on involving other communities. I’m happy to help facilitate a
cross-community discussion if we aren’t able to reach a resolution here.

Thanks,
Walaa.



On Wed, May 7, 2025 at 9:20 PM Steven Wu <st...@gmail.com> wrote:

> I agree with Dan that we shouldn't solve catalog naming in the Iceberg
> view spec. I am not convinced that the proposed change will make the table
> identifier resolution more clear and portable. The recommendation of using
> engines' current catalog and database can cause context dependent
> resolution results, which seems non-deterministic to me.
>
> Walaa, you raised a point in the doc that the current catalog resolution
> logic (default-catalog field, then view catalog) is challenging and
> unrealistic for engines (like Spark and Trino). It will be great to get
> more inputs from the broader community on this part.
>
>
> On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:
>
>> In Spark, I believe that the USE commands sets the current catalog and
>> namespace.  This affects both where the view is created and how unqualified
>> table identifiers are resolved.  I also don't see an issue with saving the
>> current catalog and namespace into the view metadata's default-catalog and
>> default-namespace fields.
>>
>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> > I think that's the lesser evil compared to Iceberg specifying how
>>> engines should resolve identifiers
>>>
>>> I think this is also similar to the previous point. It is the other way
>>> around. Right now the spec dictates how to resolve (through employing a
>>> view-specific `default-catalog` field). The proposal is suggesting to get
>>> out of this space and let engines handle it similar to how they handle all
>>> identifiers.
>>>
>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> > I thought "default-catalog" could be set via the USE command.
>>>>
>>>> Benny, I think this is a misconception or miscommunication. The USE
>>>> command has no impact on the `default-catalog` field. In fact, the
>>>> proposal's direction is exactly to establish that USE command should
>>>> influence how tables are resolved, same like everywhere else. Right now it
>>>> is not the case under the current spec.
>>>>
>>>>
>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com> wrote:
>>>>
>>>>> > there is no SQL construct today to explicitly set default-catalog
>>>>>
>>>>> I thought "default-catalog" could be set via the USE command.
>>>>>
>>>>> I generally agree with Dan about requiring consistent catalog names.
>>>>> I think that's the lesser evil compared to Iceberg specifying how engines
>>>>> should resolve identifiers.  Another thing to consider is that identifier
>>>>> resolution can be very expensive at query validation time if identifiers
>>>>> need to be looked up from a bunch of places.  Hopefully, it should be
>>>>> possible to define a view in such a way that identifiers can be resolved on
>>>>> the first try.
>>>>>
>>>>> Benny
>>>>>
>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> Hi Rishabh,
>>>>>>
>>>>>> You're right that the proposal touches on two aspects, and resolution
>>>>>> rules are one of them. The other aspect is the proposal's position that
>>>>>> table identifiers should be stored in metadata exactly as they appear in
>>>>>> the view text (e.g., even if they're two-part or partially qualified),
>>>>>> along with their corresponding UUIDs for validation. This applies both to
>>>>>> referenced input tables and the storage table identifier in materialized
>>>>>> views.
>>>>>>
>>>>>> We may be able to converge on this storage format even if we haven't
>>>>>> yet converged on the resolution fallback rules. I believe both resolution
>>>>>> strategies currently being discussed would still lead to storing
>>>>>> identifiers in this way.
>>>>>>
>>>>>> I'm supportive of moving forward with consensus on the identifier
>>>>>> storage format. That said, we may continue to run into questions related to
>>>>>> resolution during implementation. For example: Should the storage table
>>>>>> identifier follow the same default-catalog and default-namespace resolution
>>>>>> behavior as other table references?
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa.
>>>>>>
>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>>
>>>>>>> Hello Walaa,
>>>>>>>
>>>>>>> Thanks for starting this discussion.
>>>>>>>
>>>>>>> I think we should decouple at least the MV Spec from the proposal to
>>>>>>> change the current behavior of view resolution.
>>>>>>>
>>>>>>> We can continue having the discussion if the current view spec needs
>>>>>>> to be changed or not. Based on the decision at a later point if required we
>>>>>>> can update the view resolution rule.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rishabh
>>>>>>>
>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Correction of typo: both engines seem to set default-catalog to the
>>>>>>>> view catalog if it is defined, or to null if the view catalog is not
>>>>>>>> defined.
>>>>>>>>
>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Dan,
>>>>>>>>>
>>>>>>>>> Thanks again for your response.
>>>>>>>>>
>>>>>>>>> I agree that catalog renaming is an environmental event, but it's
>>>>>>>>> a real one that happens frequently in practice.
>>>>>>>>> Saying that the Iceberg spec cannot accommodate something as
>>>>>>>>> common as catalog renaming feels very restrictive, and could make the spec
>>>>>>>>> less practical, even unusable, for real-world deployments.
>>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>>
>>>>>>>>> Further, it's worth noting that the table spec is resilient to
>>>>>>>>> catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>>> Both specifications are deterministic in their definition, but one
>>>>>>>>> is more fragile to environmental changes than the other. Improving
>>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>>> more portable over time.
>>>>>>>>>
>>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>>> Today, both engines seem to set default-catalog to null if the
>>>>>>>>> view catalog is defined, or to the view catalog if not.
>>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Walaa
>>>>>>>>>
>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dw...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Walaa,
>>>>>>>>>>
>>>>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>>>>
>>>>>>>>>> This problem stems from the exact environmental/configuration
>>>>>>>>>> issue that we should not be trying to address.  I don't think we would
>>>>>>>>>> expect references to survive a catalog rename.  That's not something
>>>>>>>>>> covered by the spec and needs to be handled separately as a platform-level
>>>>>>>>>> migration specific to the affected environment.
>>>>>>>>>>
>>>>>>>>>> The identifier resolution logic is clear and deterministic.  It
>>>>>>>>>> should not matter whether an engine resolves and encodes the
>>>>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>>>>
>>>>>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>>>>>> something we should be trying to define here.
>>>>>>>>>>
>>>>>>>>>> -Dan
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>>
>>>>>>>>>>> I believe the issues we’re seeing now go beyond just catalog
>>>>>>>>>>> naming consistency. The behavior around default-catalog itself introduces
>>>>>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>>>>>> For example:
>>>>>>>>>>>
>>>>>>>>>>> * When default-catalog is set to null, tables inside views
>>>>>>>>>>> remain reachable after a catalog rename. But if it is set to a non-null
>>>>>>>>>>> value, table references will break.
>>>>>>>>>>>
>>>>>>>>>>> * default-catalog causes table references inside views to be
>>>>>>>>>>> early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>>
>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the spec.
>>>>>>>>>>> There is no way to fully align Spark's behavior without making invasive
>>>>>>>>>>> changes to the Spark SQL grammar and the View DataSource API (specifically
>>>>>>>>>>> on the CREATE side). This challenge would extend to other engines too. Both
>>>>>>>>>>> Spark and Trino set this field based on a heuristic in today's
>>>>>>>>>>> implementation.
>>>>>>>>>>>
>>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>>
>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with
>>>>>>>>>>> that model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>>
>>>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>>>> Also, do you believe the Spark and Trino communities would align
>>>>>>>>>>> around having table resolution behave inconsistently between queries and
>>>>>>>>>>> views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Walaa
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <dw...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was
>>>>>>>>>>>> introduced, but I think we need to step back and align on what we are
>>>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>>>
>>>>>>>>>>>> The issues we're discussing largely stem from using multiple
>>>>>>>>>>>> engines with cross catalog references and configurations where catalog
>>>>>>>>>>>> names are not aligned.  If we have multiple engines that all have the same
>>>>>>>>>>>> catalog names/configurations, the current spec implementation is well
>>>>>>>>>>>> defined for table resolution even across catalogs.  The 'default-catalog'
>>>>>>>>>>>> (and namespace equivalent) was intended to address the resolution within
>>>>>>>>>>>> the context of the sql text, not to address catalog/naming inconsistencies.
>>>>>>>>>>>>
>>>>>>>>>>>> I feel like we're trying to adapt the original intent to
>>>>>>>>>>>> address the catalog naming/configuration and would argue that we shouldn't
>>>>>>>>>>>> attempt to do that as part of the spec.  Inconsistently named catalogs are
>>>>>>>>>>>> a reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>>
>>>>>>>>>>>> We should support and advocate for consistency in catalog
>>>>>>>>>>>> naming and define the spec along those lines.  The fact is that with all of
>>>>>>>>>>>> the recent work that's gone into making catalogs pluggable, it makes more
>>>>>>>>>>>> sense to just register catalog configuration with consistent names (even if
>>>>>>>>>>>> you have to duplicate the configuration for supporting existing
>>>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>>>
>>>>>>>>>>>> If the goal is to create clever ways to hack the late binding
>>>>>>>>>>>> resolution to swap in different catalogs or make references contextual, I
>>>>>>>>>>>> feel like that is something we should strongly discourage as it leads to
>>>>>>>>>>>> confusion about what is resolved as part of the query.
>>>>>>>>>>>>
>>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>>
>>>>>>>>>>>> -Dan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I think the intention with the "default-catalog" was that
>>>>>>>>>>>>> every query engine uses it to store its session default catalog at the time
>>>>>>>>>>>>> of creating the view. This way the view could be reused in another session.
>>>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Generally we have different environments we want to support
>>>>>>>>>>>>> with the view spec:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>>
>>>>>>>>>>>>> When the environment supports it, using consistent catalog
>>>>>>>>>>>>> names can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>>>>>> any issues.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>>
>>>>>>>>>>>>> This can be the case when different query engines refer to the
>>>>>>>>>>>>> same physical catalog by different names. This often happens because
>>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you think of introducing a view property that
>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should be used?
>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments where you can
>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly fallback to
>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields from the
>>>>>>>>>>>>> view spec completely.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level
>>>>>>>>>>>>> default catalogs or namespaces.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Instead:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>>> query time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as they
>>>>>>>>>>>>> appear in the view SQL text.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it
>>>>>>>>>>>>> should still lack a catalog in the stored metadata.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the
>>>>>>>>>>>>>> fallback creates. Also just to be clear, at this point and after clarifying
>>>>>>>>>>>>>> the current spec intentions, I am convinced that we should remove the
>>>>>>>>>>>>>> default catalog and default namespace fields altogether.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - I don't particularly like the current behavior of "default
>>>>>>>>>>>>>>> to the view's catalog when default-catalog is not set".  Fundamentally, I
>>>>>>>>>>>>>>> believe the intent of default-catalog and default-namespace is there to
>>>>>>>>>>>>>>> help users write more concise SQL.
>>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't think
>>>>>>>>>>>>>>> we should design something that says first use this catalog, then that
>>>>>>>>>>>>>>> catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no
>>>>>>>>>>>>>>> value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>>> - Interoperability between different engine types is very
>>>>>>>>>>>>>>> hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the
>>>>>>>>>>>>>>> lines of something Russell proposed last year:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We've been looking at this interoperable identifier problem
>>>>>>>>>>>>>>> through the lens of catalog resolution but maybe the right approach is
>>>>>>>>>>>>>>> really about templating.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a view
>>>>>>>>>>>>>>> to span catalogs to support non-Iceberg tables.   Also, the default-catalog
>>>>>>>>>>>>>>> property could be templated as well.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I
>>>>>>>>>>>>>>>> shared regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback should be
>>>>>>>>>>>>>>>>> the catalog
>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter what
>>>>>>>>>>>>>>>>> the catalog
>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog, this
>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog, catalog
>>>>>>>>>>>>>>>>> names
>>>>>>>>>>>>>>>>> should be included in the reference name already. So no
>>>>>>>>>>>>>>>>> ambiguity
>>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>>>> catalog should be named consistently across usage for
>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the
>>>>>>>>>>>>>>>>> current fallback
>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for not
>>>>>>>>>>>>>>>>> conforming
>>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to
>>>>>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view
>>>>>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist before its
>>>>>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible in Spark’s
>>>>>>>>>>>>>>>>> implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting
>>>>>>>>>>>>>>>>> would require extending SQL syntax and engine APIs to promote it to a
>>>>>>>>>>>>>>>>> first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time
>>>>>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding model between
>>>>>>>>>>>>>>>>> tables vs views (early vs late), and between tables in views and in queries
>>>>>>>>>>>>>>>>> (again early vs late). For example, views and tables in queries can
>>>>>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when they are used
>>>>>>>>>>>>>>>>> inside views -- it even applies to views inside views, which makes this
>>>>>>>>>>>>>>>>> very hard to reason about considering nesting.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW statement
>>>>>>>>>>>>>>>>> the query engine knowns which catalog the view is being created in. So even
>>>>>>>>>>>>>>>>> though we typically use late binding to resolve the view catalog at query
>>>>>>>>>>>>>>>>> time, it can also be used at creation time.
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the "view
>>>>>>>>>>>>>>>>> catalog" where the view is going to be created in. It can use that catalog
>>>>>>>>>>>>>>>>> to resolve partial table identifiers if "default-catalog" is not set.
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where partial
>>>>>>>>>>>>>>>>> identifiers in the view query resolve to a different catalog compared to
>>>>>>>>>>>>>>>>> using them outside of a view.
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>>>>> "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the tables
>>>>>>>>>>>>>>>>> referenced in the view exist, how can we do that when default-catalog isn't
>>>>>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet?
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we
>>>>>>>>>>>>>>>>> validate that a table exists without a default catalog, or do we always use
>>>>>>>>>>>>>>>>> the current session catalog?
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding. Just to
>>>>>>>>>>>>>>>>> clarify: when I referred to late binding as “similar” to the proposal, I
>>>>>>>>>>>>>>>>> was acknowledging the distinction between view-level and table-level
>>>>>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting question
>>>>>>>>>>>>>>>>> and a potential gap: if default-catalog is only defined at query time, how
>>>>>>>>>>>>>>>>> should resolution work during view creation? For example, if we want to
>>>>>>>>>>>>>>>>> validate that the tables referenced in the view exist, how can we do that
>>>>>>>>>>>>>>>>> when default-catalog isn't defined, since the view hasn't been created or
>>>>>>>>>>>>>>>>> loaded yet?
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view catalog
>>>>>>>>>>>>>>>>> is resolved at query time.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish
>>>>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>>>>>> still work.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late
>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension to all partial
>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. Resolving the view
>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the view metadata in a
>>>>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced in
>>>>>>>>>>>>>>>>> a SQL statement. That statement contains the information for the query
>>>>>>>>>>>>>>>>> engine to resolve the catalog of the view.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT * FROM
>>>>>>>>>>>>>>>>> catalogB.namespace.view (after renaming catalogA to catalogB, and keeping
>>>>>>>>>>>>>>>>> everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to
>>>>>>>>>>>>>>>>> me like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of
>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec:
>>>>>>>>>>>>>>>>> The view is executed when it's being referenced in a SQL statement. That
>>>>>>>>>>>>>>>>> statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because the
>>>>>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For example
>>>>>>>>>>>>>>>>> for the following statement:
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS SELECT
>>>>>>>>>>>>>>>>> * from sales.orders;
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA",
>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same as just
>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This is because
>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of "sales.orders" would
>>>>>>>>>>>>>>>>> default to "catalogA".
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view spec,
>>>>>>>>>>>>>>>>> because it has the "closure" property. Because of the fact that the "view
>>>>>>>>>>>>>>>>> catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point before
>>>>>>>>>>>>>>>>> going deeper:
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback
>>>>>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table identifiers alone,
>>>>>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed fallback to the
>>>>>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help
>>>>>>>>>>>>>>>>> clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog”
>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if default-catalog is
>>>>>>>>>>>>>>>>> null?
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we can’t
>>>>>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then defaulting to
>>>>>>>>>>>>>>>>> it is problematic. And if views can't be created in the default catalog,
>>>>>>>>>>>>>>>>> then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the current
>>>>>>>>>>>>>>>>> session setup is in my opinion the definition of "non-determinism". You
>>>>>>>>>>>>>>>>> could be running the same query-engine and catalog-setup on different days,
>>>>>>>>>>>>>>>>> with different default session catalogs (which is rather common), and would
>>>>>>>>>>>>>>>>> be getting different results.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view always
>>>>>>>>>>>>>>>>> produces the same results. The current behavior has some rough edges in
>>>>>>>>>>>>>>>>> very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup
>>>>>>>>>>>>>>>>> more robust. Which is great. However, having UUID validation still requires
>>>>>>>>>>>>>>>>> us to have a portable table identifier specification. Even if we have the
>>>>>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply isn't an
>>>>>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog interface is defined
>>>>>>>>>>>>>>>>> in terms of table identifiers.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and
>>>>>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for
>>>>>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving
>>>>>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text with table
>>>>>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use the catalog
>>>>>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret or use.
>>>>>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated catalog that
>>>>>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In such
>>>>>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in `my_catalog.namespace1.table1`)
>>>>>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to underlying
>>>>>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the catalog
>>>>>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs actually behave
>>>>>>>>>>>>>>>>> in practice.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have some
>>>>>>>>>>>>>>>>> concerns with resolving catalog names on the fly with query engine defined
>>>>>>>>>>>>>>>>> catalog names. This introduces some flexibility at first glance, but also
>>>>>>>>>>>>>>>>> makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and
>>>>>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a vote next
>>>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The
>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog
>>>>>>>>>>>>>>>>> names in the spec refer to logical catalogs—i.e., the first part of a
>>>>>>>>>>>>>>>>> three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think view
>>>>>>>>>>>>>>>>> portability is a very important topic for us to continue discussing as it
>>>>>>>>>>>>>>>>> relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may
>>>>>>>>>>>>>>>>> impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've
>>>>>>>>>>>>>>>>> added a few comments to get a better understanding of how this will look
>>>>>>>>>>>>>>>>> like in the actual implementation.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin
>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our discussion
>>>>>>>>>>>>>>>>> on how to reference table identifiers from Iceberg metadata, a key aspect
>>>>>>>>>>>>>>>>> of the view specification, particularly in relation to the MV (materialized
>>>>>>>>>>>>>>>>> view) extensions.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a few
>>>>>>>>>>>>>>>>> community members to better understand how the current spec is being
>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look
>>>>>>>>>>>>>>>>> forward to your feedback and working together to move this forward so we
>>>>>>>>>>>>>>>>> can finalize the MV spec as well.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Steven Wu <st...@gmail.com>]

I agree with Dan that we shouldn't solve catalog naming in the Iceberg view
spec. I am not convinced that the proposed change will make the table
identifier resolution more clear and portable. The recommendation of using
engines' current catalog and database can cause context dependent
resolution results, which seems non-deterministic to me.

Walaa, you raised a point in the doc that the current catalog resolution
logic (default-catalog field, then view catalog) is challenging and
unrealistic for engines (like Spark and Trino). It will be great to get
more inputs from the broader community on this part.


On Tue, May 6, 2025 at 9:21 AM Benny Chow <bt...@gmail.com> wrote:

> In Spark, I believe that the USE commands sets the current catalog and
> namespace.  This affects both where the view is created and how unqualified
> table identifiers are resolved.  I also don't see an issue with saving the
> current catalog and namespace into the view metadata's default-catalog and
> default-namespace fields.
>
> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> > I think that's the lesser evil compared to Iceberg specifying how
>> engines should resolve identifiers
>>
>> I think this is also similar to the previous point. It is the other way
>> around. Right now the spec dictates how to resolve (through employing a
>> view-specific `default-catalog` field). The proposal is suggesting to get
>> out of this space and let engines handle it similar to how they handle all
>> identifiers.
>>
>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> > I thought "default-catalog" could be set via the USE command.
>>>
>>> Benny, I think this is a misconception or miscommunication. The USE
>>> command has no impact on the `default-catalog` field. In fact, the
>>> proposal's direction is exactly to establish that USE command should
>>> influence how tables are resolved, same like everywhere else. Right now it
>>> is not the case under the current spec.
>>>
>>>
>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com> wrote:
>>>
>>>> > there is no SQL construct today to explicitly set default-catalog
>>>>
>>>> I thought "default-catalog" could be set via the USE command.
>>>>
>>>> I generally agree with Dan about requiring consistent catalog names.  I
>>>> think that's the lesser evil compared to Iceberg specifying how engines
>>>> should resolve identifiers.  Another thing to consider is that identifier
>>>> resolution can be very expensive at query validation time if identifiers
>>>> need to be looked up from a bunch of places.  Hopefully, it should be
>>>> possible to define a view in such a way that identifiers can be resolved on
>>>> the first try.
>>>>
>>>> Benny
>>>>
>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> Hi Rishabh,
>>>>>
>>>>> You're right that the proposal touches on two aspects, and resolution
>>>>> rules are one of them. The other aspect is the proposal's position that
>>>>> table identifiers should be stored in metadata exactly as they appear in
>>>>> the view text (e.g., even if they're two-part or partially qualified),
>>>>> along with their corresponding UUIDs for validation. This applies both to
>>>>> referenced input tables and the storage table identifier in materialized
>>>>> views.
>>>>>
>>>>> We may be able to converge on this storage format even if we haven't
>>>>> yet converged on the resolution fallback rules. I believe both resolution
>>>>> strategies currently being discussed would still lead to storing
>>>>> identifiers in this way.
>>>>>
>>>>> I'm supportive of moving forward with consensus on the identifier
>>>>> storage format. That said, we may continue to run into questions related to
>>>>> resolution during implementation. For example: Should the storage table
>>>>> identifier follow the same default-catalog and default-namespace resolution
>>>>> behavior as other table references?
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>>> bhatiarishabh63@gmail.com> wrote:
>>>>>
>>>>>> Hello Walaa,
>>>>>>
>>>>>> Thanks for starting this discussion.
>>>>>>
>>>>>> I think we should decouple at least the MV Spec from the proposal to
>>>>>> change the current behavior of view resolution.
>>>>>>
>>>>>> We can continue having the discussion if the current view spec needs
>>>>>> to be changed or not. Based on the decision at a later point if required we
>>>>>> can update the view resolution rule.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Rishabh
>>>>>>
>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> Correction of typo: both engines seem to set default-catalog to the
>>>>>>> view catalog if it is defined, or to null if the view catalog is not
>>>>>>> defined.
>>>>>>>
>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Dan,
>>>>>>>>
>>>>>>>> Thanks again for your response.
>>>>>>>>
>>>>>>>> I agree that catalog renaming is an environmental event, but it's a
>>>>>>>> real one that happens frequently in practice.
>>>>>>>> Saying that the Iceberg spec cannot accommodate something as common
>>>>>>>> as catalog renaming feels very restrictive, and could make the spec less
>>>>>>>> practical, even unusable, for real-world deployments.
>>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>>
>>>>>>>> Further, it's worth noting that the table spec is resilient to
>>>>>>>> catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>>> Both specifications are deterministic in their definition, but one
>>>>>>>> is more fragile to environmental changes than the other. Improving
>>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>>> more portable over time.
>>>>>>>>
>>>>>>>> Separately, given that there is no SQL construct today to
>>>>>>>> explicitly set default-catalog at creation time, what is the intuition
>>>>>>>> behind how engines like Spark and Trino currently assign default-catalog?
>>>>>>>> Today, both engines seem to set default-catalog to null if the view
>>>>>>>> catalog is defined, or to the view catalog if not.
>>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Walaa
>>>>>>>>
>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dw...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Walaa,
>>>>>>>>>
>>>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>>>
>>>>>>>>> This problem stems from the exact environmental/configuration
>>>>>>>>> issue that we should not be trying to address.  I don't think we would
>>>>>>>>> expect references to survive a catalog rename.  That's not something
>>>>>>>>> covered by the spec and needs to be handled separately as a platform-level
>>>>>>>>> migration specific to the affected environment.
>>>>>>>>>
>>>>>>>>> The identifier resolution logic is clear and deterministic.  It
>>>>>>>>> should not matter whether an engine resolves and encodes the
>>>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>>>
>>>>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>>>>> something we should be trying to define here.
>>>>>>>>>
>>>>>>>>> -Dan
>>>>>>>>>
>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Dan,
>>>>>>>>>>
>>>>>>>>>> Thanks for chiming in.
>>>>>>>>>>
>>>>>>>>>> I believe the issues we’re seeing now go beyond just catalog
>>>>>>>>>> naming consistency. The behavior around default-catalog itself introduces
>>>>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>>>>> For example:
>>>>>>>>>>
>>>>>>>>>> * When default-catalog is set to null, tables inside views remain
>>>>>>>>>> reachable after a catalog rename. But if it is set to a non-null value,
>>>>>>>>>> table references will break.
>>>>>>>>>>
>>>>>>>>>> * default-catalog causes table references inside views to be
>>>>>>>>>> early bound (i.e., bound at view creation time, especially when using a
>>>>>>>>>> non-null value), while table references inside standalone queries are late
>>>>>>>>>> bound (bound at query time). This creates inconsistencies when resolving
>>>>>>>>>> the same table name inside and outside views, even within the same job.
>>>>>>>>>>
>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the spec.
>>>>>>>>>> There is no way to fully align Spark's behavior without making invasive
>>>>>>>>>> changes to the Spark SQL grammar and the View DataSource API (specifically
>>>>>>>>>> on the CREATE side). This challenge would extend to other engines too. Both
>>>>>>>>>> Spark and Trino set this field based on a heuristic in today's
>>>>>>>>>> implementation.
>>>>>>>>>>
>>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>>
>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with that
>>>>>>>>>> model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>>
>>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>>> Also, do you believe the Spark and Trino communities would align
>>>>>>>>>> around having table resolution behave inconsistently between queries and
>>>>>>>>>> views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Walaa
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <dw...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was
>>>>>>>>>>> introduced, but I think we need to step back and align on what we are
>>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>>
>>>>>>>>>>> The issues we're discussing largely stem from using multiple
>>>>>>>>>>> engines with cross catalog references and configurations where catalog
>>>>>>>>>>> names are not aligned.  If we have multiple engines that all have the same
>>>>>>>>>>> catalog names/configurations, the current spec implementation is well
>>>>>>>>>>> defined for table resolution even across catalogs.  The 'default-catalog'
>>>>>>>>>>> (and namespace equivalent) was intended to address the resolution within
>>>>>>>>>>> the context of the sql text, not to address catalog/naming inconsistencies.
>>>>>>>>>>>
>>>>>>>>>>> I feel like we're trying to adapt the original intent to address
>>>>>>>>>>> the catalog naming/configuration and would argue that we shouldn't attempt
>>>>>>>>>>> to do that as part of the spec.  Inconsistently named catalogs are a
>>>>>>>>>>> reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>>
>>>>>>>>>>> We should support and advocate for consistency in catalog naming
>>>>>>>>>>> and define the spec along those lines.  The fact is that with all of the
>>>>>>>>>>> recent work that's gone into making catalogs pluggable, it makes more sense
>>>>>>>>>>> to just register catalog configuration with consistent names (even if you
>>>>>>>>>>> have to duplicate the configuration for supporting existing
>>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>>
>>>>>>>>>>> If the goal is to create clever ways to hack the late binding
>>>>>>>>>>> resolution to swap in different catalogs or make references contextual, I
>>>>>>>>>>> feel like that is something we should strongly discourage as it leads to
>>>>>>>>>>> confusion about what is resolved as part of the query.
>>>>>>>>>>>
>>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>>
>>>>>>>>>>> -Dan
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I think the intention with the "default-catalog" was that every
>>>>>>>>>>>> query engine uses it to store its session default catalog at the time of
>>>>>>>>>>>> creating the view. This way the view could be reused in another session.
>>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>>> default-catalog.
>>>>>>>>>>>>
>>>>>>>>>>>> Generally we have different environments we want to support
>>>>>>>>>>>> with the view spec:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>>
>>>>>>>>>>>> When the environment supports it, using consistent catalog
>>>>>>>>>>>> names can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>>>>> any issues.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>>
>>>>>>>>>>>> This can be the case when different query engines refer to the
>>>>>>>>>>>> same physical catalog by different names. This often happens because
>>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> What do you think of introducing a view property that specifies
>>>>>>>>>>>> if the "default-catalog" or the "view catalog" should be used? This way,
>>>>>>>>>>>> you could use the "default-catalog" in environments where you can guarantee
>>>>>>>>>>>> consistent naming, but you would be able to directly fallback to the
>>>>>>>>>>>> "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> Jan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>>
>>>>>>>>>>>> * Remove default-catalog and default-namespace fields from the
>>>>>>>>>>>> view spec completely.
>>>>>>>>>>>>
>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level
>>>>>>>>>>>> default catalogs or namespaces.
>>>>>>>>>>>>
>>>>>>>>>>>> Instead:
>>>>>>>>>>>>
>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog
>>>>>>>>>>>> qualifier, engines should resolve it using the current engine catalog at
>>>>>>>>>>>> query time.
>>>>>>>>>>>>
>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as they
>>>>>>>>>>>> appear in the view SQL text.
>>>>>>>>>>>>
>>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it
>>>>>>>>>>>> should still lack a catalog in the stored metadata.
>>>>>>>>>>>>
>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the
>>>>>>>>>>>>> fallback creates. Also just to be clear, at this point and after clarifying
>>>>>>>>>>>>> the current spec intentions, I am convinced that we should remove the
>>>>>>>>>>>>> default catalog and default namespace fields altogether.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - I don't particularly like the current behavior of "default
>>>>>>>>>>>>>> to the view's catalog when default-catalog is not set".  Fundamentally, I
>>>>>>>>>>>>>> believe the intent of default-catalog and default-namespace is there to
>>>>>>>>>>>>>> help users write more concise SQL.
>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't think
>>>>>>>>>>>>>> we should design something that says first use this catalog, then that
>>>>>>>>>>>>>> catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no
>>>>>>>>>>>>>> value in putting views in the catalog to share with other engines
>>>>>>>>>>>>>> - Interoperability between different engine types is very
>>>>>>>>>>>>>> hard due to dialect issues... so I think we should focus on supporting
>>>>>>>>>>>>>> different clusters of the same engine type on a shared catalog.  For
>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views in a REST
>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the
>>>>>>>>>>>>>> lines of something Russell proposed last year:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We've been looking at this interoperable identifier problem
>>>>>>>>>>>>>> through the lens of catalog resolution but maybe the right approach is
>>>>>>>>>>>>>> really about templating.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a view
>>>>>>>>>>>>>> to span catalogs to support non-Iceberg tables.   Also, the default-catalog
>>>>>>>>>>>>>> property could be templated as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I
>>>>>>>>>>>>>>> shared regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback should be
>>>>>>>>>>>>>>>> the catalog
>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter what
>>>>>>>>>>>>>>>> the catalog
>>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog, this
>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog, catalog
>>>>>>>>>>>>>>>> names
>>>>>>>>>>>>>>>> should be included in the reference name already. So no
>>>>>>>>>>>>>>>> ambiguity
>>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>>> catalog should be named consistently across usage for better
>>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to
>>>>>>>>>>>>>>>> engine's session
>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the
>>>>>>>>>>>>>>>> current fallback
>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this idea.
>>>>>>>>>>>>>>>> Spark resolves table identifiers during view creation using the session’s
>>>>>>>>>>>>>>>> default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for not
>>>>>>>>>>>>>>>> conforming
>>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to
>>>>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view
>>>>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist before its
>>>>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible in Spark’s
>>>>>>>>>>>>>>>> implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting
>>>>>>>>>>>>>>>> would require extending SQL syntax and engine APIs to promote it to a
>>>>>>>>>>>>>>>> first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time
>>>>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding model between
>>>>>>>>>>>>>>>> tables vs views (early vs late), and between tables in views and in queries
>>>>>>>>>>>>>>>> (again early vs late). For example, views and tables in queries can
>>>>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when they are used
>>>>>>>>>>>>>>>> inside views -- it even applies to views inside views, which makes this
>>>>>>>>>>>>>>>> very hard to reason about considering nesting.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW statement
>>>>>>>>>>>>>>>> the query engine knowns which catalog the view is being created in. So even
>>>>>>>>>>>>>>>> though we typically use late binding to resolve the view catalog at query
>>>>>>>>>>>>>>>> time, it can also be used at creation time.
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the "view
>>>>>>>>>>>>>>>> catalog" where the view is going to be created in. It can use that catalog
>>>>>>>>>>>>>>>> to resolve partial table identifiers if "default-catalog" is not set.
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where partial
>>>>>>>>>>>>>>>> identifiers in the view query resolve to a different catalog compared to
>>>>>>>>>>>>>>>> using them outside of a view.
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>>>> "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the tables
>>>>>>>>>>>>>>>> referenced in the view exist, how can we do that when default-catalog isn't
>>>>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet?
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we
>>>>>>>>>>>>>>>> validate that a table exists without a default catalog, or do we always use
>>>>>>>>>>>>>>>> the current session catalog?
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> I think we still share the same understanding. Just to
>>>>>>>>>>>>>>>> clarify: when I referred to late binding as “similar” to the proposal, I
>>>>>>>>>>>>>>>> was acknowledging the distinction between view-level and table-level
>>>>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting question
>>>>>>>>>>>>>>>> and a potential gap: if default-catalog is only defined at query time, how
>>>>>>>>>>>>>>>> should resolution work during view creation? For example, if we want to
>>>>>>>>>>>>>>>> validate that the tables referenced in the view exist, how can we do that
>>>>>>>>>>>>>>>> when default-catalog isn't defined, since the view hasn't been created or
>>>>>>>>>>>>>>>> loaded yet?
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view catalog
>>>>>>>>>>>>>>>> is resolved at query time.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish
>>>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>>>>> still work.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late binding
>>>>>>>>>>>>>>>> applies to the view catalog name and by extension to all partial table
>>>>>>>>>>>>>>>> references when no "default-catalog" is present. Resolving the view catalog
>>>>>>>>>>>>>>>> name at query time is not opposed to storing the view metadata in a catalog.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced in a
>>>>>>>>>>>>>>>> SQL statement. That statement contains the information for the query engine
>>>>>>>>>>>>>>>> to resolve the catalog of the view.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT * FROM
>>>>>>>>>>>>>>>> catalogB.namespace.view (after renaming catalogA to catalogB, and keeping
>>>>>>>>>>>>>>>> everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to me
>>>>>>>>>>>>>>>> like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of non-determinism.
>>>>>>>>>>>>>>>> Let me try to address your questions.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec: The
>>>>>>>>>>>>>>>> view is executed when it's being referenced in a SQL statement. That
>>>>>>>>>>>>>>>> statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because the
>>>>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For example
>>>>>>>>>>>>>>>> for the following statement:
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>>>> "catalogA".
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view spec,
>>>>>>>>>>>>>>>> because it has the "closure" property. Because of the fact that the "view
>>>>>>>>>>>>>>>> catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point before
>>>>>>>>>>>>>>>> going deeper:
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback
>>>>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table identifiers alone,
>>>>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed fallback to the
>>>>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help
>>>>>>>>>>>>>>>> clarify two things in the context of the current spec:
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog” stored,
>>>>>>>>>>>>>>>> so that an engine knows to fall back to it if default-catalog is null?
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the
>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a catalog) in the
>>>>>>>>>>>>>>>> current Iceberg spec?
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> These questions are important because if we can’t
>>>>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then defaulting to
>>>>>>>>>>>>>>>> it is problematic. And if views can't be created in the default catalog,
>>>>>>>>>>>>>>>> then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the current
>>>>>>>>>>>>>>>> session setup is in my opinion the definition of "non-determinism". You
>>>>>>>>>>>>>>>> could be running the same query-engine and catalog-setup on different days,
>>>>>>>>>>>>>>>> with different default session catalogs (which is rather common), and would
>>>>>>>>>>>>>>>> be getting different results.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view always
>>>>>>>>>>>>>>>> produces the same results. The current behavior has some rough edges in
>>>>>>>>>>>>>>>> very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup
>>>>>>>>>>>>>>>> more robust. Which is great. However, having UUID validation still requires
>>>>>>>>>>>>>>>> us to have a portable table identifier specification. Even if we have the
>>>>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply isn't an
>>>>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog interface is defined
>>>>>>>>>>>>>>>> in terms of table identifiers.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and
>>>>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for
>>>>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving
>>>>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text with table
>>>>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use the catalog
>>>>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret or use.
>>>>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated catalog that
>>>>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In such
>>>>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in `my_catalog.namespace1.table1`)
>>>>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to underlying
>>>>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the catalog
>>>>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs actually behave
>>>>>>>>>>>>>>>> in practice.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have some
>>>>>>>>>>>>>>>> concerns with resolving catalog names on the fly with query engine defined
>>>>>>>>>>>>>>>> catalog names. This introduces some flexibility at first glance, but also
>>>>>>>>>>>>>>>> makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and
>>>>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a vote next
>>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The proposal
>>>>>>>>>>>>>>>> outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog
>>>>>>>>>>>>>>>> names in the spec refer to logical catalogs—i.e., the first part of a
>>>>>>>>>>>>>>>> three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think view
>>>>>>>>>>>>>>>> portability is a very important topic for us to continue discussing as it
>>>>>>>>>>>>>>>> relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may
>>>>>>>>>>>>>>>> impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've
>>>>>>>>>>>>>>>> added a few comments to get a better understanding of how this will look
>>>>>>>>>>>>>>>> like in the actual implementation.
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin
>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our discussion
>>>>>>>>>>>>>>>> on how to reference table identifiers from Iceberg metadata, a key aspect
>>>>>>>>>>>>>>>> of the view specification, particularly in relation to the MV (materialized
>>>>>>>>>>>>>>>> view) extensions.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a few
>>>>>>>>>>>>>>>> community members to better understand how the current spec is being
>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look
>>>>>>>>>>>>>>>> forward to your feedback and working together to move this forward so we
>>>>>>>>>>>>>>>> can finalize the MV spec as well.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Benny Chow <bt...@gmail.com>]

In Spark, I believe that the USE commands sets the current catalog and
namespace.  This affects both where the view is created and how unqualified
table identifiers are resolved.  I also don't see an issue with saving the
current catalog and namespace into the view metadata's default-catalog and
default-namespace fields.

On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> > I think that's the lesser evil compared to Iceberg specifying how
> engines should resolve identifiers
>
> I think this is also similar to the previous point. It is the other way
> around. Right now the spec dictates how to resolve (through employing a
> view-specific `default-catalog` field). The proposal is suggesting to get
> out of this space and let engines handle it similar to how they handle all
> identifiers.
>
> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> > I thought "default-catalog" could be set via the USE command.
>>
>> Benny, I think this is a misconception or miscommunication. The USE
>> command has no impact on the `default-catalog` field. In fact, the
>> proposal's direction is exactly to establish that USE command should
>> influence how tables are resolved, same like everywhere else. Right now it
>> is not the case under the current spec.
>>
>>
>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com> wrote:
>>
>>> > there is no SQL construct today to explicitly set default-catalog
>>>
>>> I thought "default-catalog" could be set via the USE command.
>>>
>>> I generally agree with Dan about requiring consistent catalog names.  I
>>> think that's the lesser evil compared to Iceberg specifying how engines
>>> should resolve identifiers.  Another thing to consider is that identifier
>>> resolution can be very expensive at query validation time if identifiers
>>> need to be looked up from a bunch of places.  Hopefully, it should be
>>> possible to define a view in such a way that identifiers can be resolved on
>>> the first try.
>>>
>>> Benny
>>>
>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> Hi Rishabh,
>>>>
>>>> You're right that the proposal touches on two aspects, and resolution
>>>> rules are one of them. The other aspect is the proposal's position that
>>>> table identifiers should be stored in metadata exactly as they appear in
>>>> the view text (e.g., even if they're two-part or partially qualified),
>>>> along with their corresponding UUIDs for validation. This applies both to
>>>> referenced input tables and the storage table identifier in materialized
>>>> views.
>>>>
>>>> We may be able to converge on this storage format even if we haven't
>>>> yet converged on the resolution fallback rules. I believe both resolution
>>>> strategies currently being discussed would still lead to storing
>>>> identifiers in this way.
>>>>
>>>> I'm supportive of moving forward with consensus on the identifier
>>>> storage format. That said, we may continue to run into questions related to
>>>> resolution during implementation. For example: Should the storage table
>>>> identifier follow the same default-catalog and default-namespace resolution
>>>> behavior as other table references?
>>>>
>>>> Thanks,
>>>> Walaa.
>>>>
>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>>> bhatiarishabh63@gmail.com> wrote:
>>>>
>>>>> Hello Walaa,
>>>>>
>>>>> Thanks for starting this discussion.
>>>>>
>>>>> I think we should decouple at least the MV Spec from the proposal to
>>>>> change the current behavior of view resolution.
>>>>>
>>>>> We can continue having the discussion if the current view spec needs
>>>>> to be changed or not. Based on the decision at a later point if required we
>>>>> can update the view resolution rule.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Rishabh
>>>>>
>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> Correction of typo: both engines seem to set default-catalog to the
>>>>>> view catalog if it is defined, or to null if the view catalog is not
>>>>>> defined.
>>>>>>
>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Dan,
>>>>>>>
>>>>>>> Thanks again for your response.
>>>>>>>
>>>>>>> I agree that catalog renaming is an environmental event, but it's a
>>>>>>> real one that happens frequently in practice.
>>>>>>> Saying that the Iceberg spec cannot accommodate something as common
>>>>>>> as catalog renaming feels very restrictive, and could make the spec less
>>>>>>> practical, even unusable, for real-world deployments.
>>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>>
>>>>>>> Further, it's worth noting that the table spec is resilient to
>>>>>>> catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>>> Both specifications are deterministic in their definition, but one
>>>>>>> is more fragile to environmental changes than the other. Improving
>>>>>>> resilience does not sacrifice determinism. It simply makes views safer and
>>>>>>> more portable over time.
>>>>>>>
>>>>>>> Separately, given that there is no SQL construct today to explicitly
>>>>>>> set default-catalog at creation time, what is the intuition behind how
>>>>>>> engines like Spark and Trino currently assign default-catalog?
>>>>>>> Today, both engines seem to set default-catalog to null if the view
>>>>>>> catalog is defined, or to the view catalog if not.
>>>>>>> What was the intended thought process behind this behavior?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Walaa
>>>>>>>
>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dw...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Walaa,
>>>>>>>>
>>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>>
>>>>>>>> This problem stems from the exact environmental/configuration issue
>>>>>>>> that we should not be trying to address.  I don't think we would expect
>>>>>>>> references to survive a catalog rename.  That's not something covered by
>>>>>>>> the spec and needs to be handled separately as a platform-level migration
>>>>>>>> specific to the affected environment.
>>>>>>>>
>>>>>>>> The identifier resolution logic is clear and deterministic.  It
>>>>>>>> should not matter whether an engine resolves and encodes the
>>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>>
>>>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>>>> something we should be trying to define here.
>>>>>>>>
>>>>>>>> -Dan
>>>>>>>>
>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Dan,
>>>>>>>>>
>>>>>>>>> Thanks for chiming in.
>>>>>>>>>
>>>>>>>>> I believe the issues we’re seeing now go beyond just catalog
>>>>>>>>> naming consistency. The behavior around default-catalog itself introduces
>>>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>>>> For example:
>>>>>>>>>
>>>>>>>>> * When default-catalog is set to null, tables inside views remain
>>>>>>>>> reachable after a catalog rename. But if it is set to a non-null value,
>>>>>>>>> table references will break.
>>>>>>>>>
>>>>>>>>> * default-catalog causes table references inside views to be early
>>>>>>>>> bound (i.e., bound at view creation time, especially when using a non-null
>>>>>>>>> value), while table references inside standalone queries are late bound
>>>>>>>>> (bound at query time). This creates inconsistencies when resolving the same
>>>>>>>>> table name inside and outside views, even within the same job.
>>>>>>>>>
>>>>>>>>> * It causes Spark's and Trino behavior to drift from the spec.
>>>>>>>>> There is no way to fully align Spark's behavior without making invasive
>>>>>>>>> changes to the Spark SQL grammar and the View DataSource API (specifically
>>>>>>>>> on the CREATE side). This challenge would extend to other engines too. Both
>>>>>>>>> Spark and Trino set this field based on a heuristic in today's
>>>>>>>>> implementation.
>>>>>>>>>
>>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>>
>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with that
>>>>>>>>> model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>>
>>>>>>>>> How would you suggest approaching the engine-level changes
>>>>>>>>> required to support the current default-catalog field?
>>>>>>>>> Also, do you believe the Spark and Trino communities would align
>>>>>>>>> around having table resolution behave inconsistently between queries and
>>>>>>>>> views, or inconsistency between Iceberg and other types of views?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Walaa
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <dw...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was
>>>>>>>>>> introduced, but I think we need to step back and align on what we are
>>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>>
>>>>>>>>>> The issues we're discussing largely stem from using multiple
>>>>>>>>>> engines with cross catalog references and configurations where catalog
>>>>>>>>>> names are not aligned.  If we have multiple engines that all have the same
>>>>>>>>>> catalog names/configurations, the current spec implementation is well
>>>>>>>>>> defined for table resolution even across catalogs.  The 'default-catalog'
>>>>>>>>>> (and namespace equivalent) was intended to address the resolution within
>>>>>>>>>> the context of the sql text, not to address catalog/naming inconsistencies.
>>>>>>>>>>
>>>>>>>>>> I feel like we're trying to adapt the original intent to address
>>>>>>>>>> the catalog naming/configuration and would argue that we shouldn't attempt
>>>>>>>>>> to do that as part of the spec.  Inconsistently named catalogs are a
>>>>>>>>>> reality, but we should consider that a configuration/environmental issue,
>>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>>
>>>>>>>>>> We should support and advocate for consistency in catalog naming
>>>>>>>>>> and define the spec along those lines.  The fact is that with all of the
>>>>>>>>>> recent work that's gone into making catalogs pluggable, it makes more sense
>>>>>>>>>> to just register catalog configuration with consistent names (even if you
>>>>>>>>>> have to duplicate the configuration for supporting existing
>>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>>
>>>>>>>>>> If the goal is to create clever ways to hack the late binding
>>>>>>>>>> resolution to swap in different catalogs or make references contextual, I
>>>>>>>>>> feel like that is something we should strongly discourage as it leads to
>>>>>>>>>> confusion about what is resolved as part of the query.
>>>>>>>>>>
>>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>>
>>>>>>>>>> -Dan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> I think the intention with the "default-catalog" was that every
>>>>>>>>>>> query engine uses it to store its session default catalog at the time of
>>>>>>>>>>> creating the view. This way the view could be reused in another session.
>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>>> default-catalog.
>>>>>>>>>>>
>>>>>>>>>>> Generally we have different environments we want to support with
>>>>>>>>>>> the view spec:
>>>>>>>>>>>
>>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>>
>>>>>>>>>>> When the environment supports it, using consistent catalog names
>>>>>>>>>>> can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>>>> any issues.
>>>>>>>>>>>
>>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>>
>>>>>>>>>>> This can be the case when different query engines refer to the
>>>>>>>>>>> same physical catalog by different names. This often happens because
>>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What do you think of introducing a view property that specifies
>>>>>>>>>>> if the "default-catalog" or the "view catalog" should be used? This way,
>>>>>>>>>>> you could use the "default-catalog" in environments where you can guarantee
>>>>>>>>>>> consistent naming, but you would be able to directly fallback to the
>>>>>>>>>>> "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Jan
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>
>>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>>
>>>>>>>>>>> * Remove default-catalog and default-namespace fields from the
>>>>>>>>>>> view spec completely.
>>>>>>>>>>>
>>>>>>>>>>> * Hence, we do not attempt to define separate view-level default
>>>>>>>>>>> catalogs or namespaces.
>>>>>>>>>>>
>>>>>>>>>>> Instead:
>>>>>>>>>>>
>>>>>>>>>>> * If a table identifier inside a view lacks a catalog qualifier,
>>>>>>>>>>> engines should resolve it using the current engine catalog at query time.
>>>>>>>>>>>
>>>>>>>>>>> * Reference table identifiers in the metadata exactly as they
>>>>>>>>>>> appear in the view SQL text.
>>>>>>>>>>>
>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it should
>>>>>>>>>>> still lack a catalog in the stored metadata.
>>>>>>>>>>>
>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Walaa.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the
>>>>>>>>>>>> fallback creates. Also just to be clear, at this point and after clarifying
>>>>>>>>>>>> the current spec intentions, I am convinced that we should remove the
>>>>>>>>>>>> default catalog and default namespace fields altogether.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - I don't particularly like the current behavior of "default
>>>>>>>>>>>>> to the view's catalog when default-catalog is not set".  Fundamentally, I
>>>>>>>>>>>>> believe the intent of default-catalog and default-namespace is there to
>>>>>>>>>>>>> help users write more concise SQL.
>>>>>>>>>>>>> - spark session catalog is engine specific and I don't think
>>>>>>>>>>>>> we should design something that says first use this catalog, then that
>>>>>>>>>>>>> catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no value
>>>>>>>>>>>>> in putting views in the catalog to share with other engines
>>>>>>>>>>>>> - Interoperability between different engine types is very hard
>>>>>>>>>>>>> due to dialect issues... so I think we should focus on supporting different
>>>>>>>>>>>>> clusters of the same engine type on a shared catalog.  For example, AI and
>>>>>>>>>>>>> BI clusters on Spark sharing the same views in a REST catalog.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the
>>>>>>>>>>>>> lines of something Russell proposed last year:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>>
>>>>>>>>>>>>> We've been looking at this interoperable identifier problem
>>>>>>>>>>>>> through the lens of catalog resolution but maybe the right approach is
>>>>>>>>>>>>> really about templating.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a view
>>>>>>>>>>>>> to span catalogs to support non-Iceberg tables.   Also, the default-catalog
>>>>>>>>>>>>> property could be templated as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>> Benny
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I shared
>>>>>>>>>>>>>> regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>>> not defined. Current view spec says the fallback should be
>>>>>>>>>>>>>>> the catalog
>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter what the
>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog, this
>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>>> - If a view refers to tables from another catalog, catalog
>>>>>>>>>>>>>>> names
>>>>>>>>>>>>>>> should be included in the reference name already. So no
>>>>>>>>>>>>>>> ambiguity
>>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>>> catalog should be named consistently across usage for better
>>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to engine's
>>>>>>>>>>>>>>> session
>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the current
>>>>>>>>>>>>>>> fallback
>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this idea.
>>>>>>>>>>>>>>> Spark resolves table identifiers during view creation using the session’s
>>>>>>>>>>>>>>> default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for not
>>>>>>>>>>>>>>> conforming
>>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to
>>>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view
>>>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist before its
>>>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible in Spark’s
>>>>>>>>>>>>>>> implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting
>>>>>>>>>>>>>>> would require extending SQL syntax and engine APIs to promote it to a
>>>>>>>>>>>>>>> first-class view concept. This would be intrusive, non-intuitive, and
>>>>>>>>>>>>>>> realistically very difficult to standardize across engines.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this
>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation using the
>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time
>>>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding model between
>>>>>>>>>>>>>>> tables vs views (early vs late), and between tables in views and in queries
>>>>>>>>>>>>>>> (again early vs late). For example, views and tables in queries can
>>>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when they are used
>>>>>>>>>>>>>>> inside views -- it even applies to views inside views, which makes this
>>>>>>>>>>>>>>> very hard to reason about considering nesting.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW statement
>>>>>>>>>>>>>>> the query engine knowns which catalog the view is being created in. So even
>>>>>>>>>>>>>>> though we typically use late binding to resolve the view catalog at query
>>>>>>>>>>>>>>> time, it can also be used at creation time.
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> The query engine would need to keep track of the "view
>>>>>>>>>>>>>>> catalog" where the view is going to be created in. It can use that catalog
>>>>>>>>>>>>>>> to resolve partial table identifiers if "default-catalog" is not set.
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where partial
>>>>>>>>>>>>>>> identifiers in the view query resolve to a different catalog compared to
>>>>>>>>>>>>>>> using them outside of a view.
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>>> "catalogA", which is the view's catalog.
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> For example, if we want to validate that the tables
>>>>>>>>>>>>>>> referenced in the view exist, how can we do that when default-catalog isn't
>>>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet?
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we
>>>>>>>>>>>>>>> validate that a table exists without a default catalog, or do we always use
>>>>>>>>>>>>>>> the current session catalog?
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> I think we still share the same understanding. Just to
>>>>>>>>>>>>>>> clarify: when I referred to late binding as “similar” to the proposal, I
>>>>>>>>>>>>>>> was acknowledging the distinction between view-level and table-level
>>>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> That said, this still raises an interesting question and
>>>>>>>>>>>>>>> a potential gap: if default-catalog is only defined at query time, how
>>>>>>>>>>>>>>> should resolution work during view creation? For example, if we want to
>>>>>>>>>>>>>>> validate that the tables referenced in the view exist, how can we do that
>>>>>>>>>>>>>>> when default-catalog isn't defined, since the view hasn't been created or
>>>>>>>>>>>>>>> loaded yet?
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view catalog is
>>>>>>>>>>>>>>> resolved at query time.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish
>>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>>>> still work.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late binding
>>>>>>>>>>>>>>> applies to the view catalog name and by extension to all partial table
>>>>>>>>>>>>>>> references when no "default-catalog" is present. Resolving the view catalog
>>>>>>>>>>>>>>> name at query time is not opposed to storing the view metadata in a catalog.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced in a
>>>>>>>>>>>>>>> SQL statement. That statement contains the information for the query engine
>>>>>>>>>>>>>>> to resolve the catalog of the view.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT * FROM
>>>>>>>>>>>>>>> catalogB.namespace.view (after renaming catalogA to catalogB, and keeping
>>>>>>>>>>>>>>> everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to me
>>>>>>>>>>>>>>> like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of non-determinism.
>>>>>>>>>>>>>>> Let me try to address your questions.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec: The
>>>>>>>>>>>>>>> view is executed when it's being referenced in a SQL statement. That
>>>>>>>>>>>>>>> statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because the
>>>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For example
>>>>>>>>>>>>>>> for the following statement:
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>>> "catalogA".
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view spec,
>>>>>>>>>>>>>>> because it has the "closure" property. Because of the fact that the "view
>>>>>>>>>>>>>>> catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point before
>>>>>>>>>>>>>>> going deeper:
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback
>>>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table identifiers alone,
>>>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed fallback to the
>>>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help
>>>>>>>>>>>>>>> clarify two things in the context of the current spec:
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog” stored,
>>>>>>>>>>>>>>> so that an engine knows to fall back to it if default-catalog is null?
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the session's
>>>>>>>>>>>>>>> default catalog (i.e., without specifying a catalog) in the current Iceberg
>>>>>>>>>>>>>>> spec?
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> These questions are important because if we can’t
>>>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then defaulting to
>>>>>>>>>>>>>>> it is problematic. And if views can't be created in the default catalog,
>>>>>>>>>>>>>>> then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the current
>>>>>>>>>>>>>>> session setup is in my opinion the definition of "non-determinism". You
>>>>>>>>>>>>>>> could be running the same query-engine and catalog-setup on different days,
>>>>>>>>>>>>>>> with different default session catalogs (which is rather common), and would
>>>>>>>>>>>>>>> be getting different results.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view always
>>>>>>>>>>>>>>> produces the same results. The current behavior has some rough edges in
>>>>>>>>>>>>>>> very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup
>>>>>>>>>>>>>>> more robust. Which is great. However, having UUID validation still requires
>>>>>>>>>>>>>>> us to have a portable table identifier specification. Even if we have the
>>>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply isn't an
>>>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog interface is defined
>>>>>>>>>>>>>>> in terms of table identifiers.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and
>>>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for
>>>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving
>>>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text with table
>>>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use the catalog
>>>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret or use.
>>>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated catalog that
>>>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In such
>>>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in `my_catalog.namespace1.table1`)
>>>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to underlying
>>>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the catalog
>>>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs actually behave
>>>>>>>>>>>>>>> in practice.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have some
>>>>>>>>>>>>>>> concerns with resolving catalog names on the fly with query engine defined
>>>>>>>>>>>>>>> catalog names. This introduces some flexibility at first glance, but also
>>>>>>>>>>>>>>> makes misconfiguration difficult to explain.
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin Moustafa
>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and
>>>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a vote next
>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The proposal
>>>>>>>>>>>>>>> outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog
>>>>>>>>>>>>>>> names in the spec refer to logical catalogs—i.e., the first part of a
>>>>>>>>>>>>>>> three-part identifier. These correspond to Spark's DataSourceV2 catalogs,
>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level of abstraction
>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used in the view spec.
>>>>>>>>>>>>>>> The reason is that table identifiers in the view definition/text itself
>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they interface directly
>>>>>>>>>>>>>>> with the engine and not a specific metastore).
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think view
>>>>>>>>>>>>>>> portability is a very important topic for us to continue discussing as it
>>>>>>>>>>>>>>> relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may
>>>>>>>>>>>>>>> impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
>>>>>>>>>>>>>>> Tudenhöfner <et...@apache.org> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've
>>>>>>>>>>>>>>> added a few comments to get a better understanding of how this will look
>>>>>>>>>>>>>>> like in the actual implementation.
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin
>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our discussion
>>>>>>>>>>>>>>> on how to reference table identifiers from Iceberg metadata, a key aspect
>>>>>>>>>>>>>>> of the view specification, particularly in relation to the MV (materialized
>>>>>>>>>>>>>>> view) extensions.
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a few
>>>>>>>>>>>>>>> community members to better understand how the current spec is being
>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look
>>>>>>>>>>>>>>> forward to your feedback and working together to move this forward so we
>>>>>>>>>>>>>>> can finalize the MV spec as well.
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

[Walaa Eldin Moustafa <wa...@gmail.com>]

> I think that's the lesser evil compared to Iceberg specifying how engines
should resolve identifiers

I think this is also similar to the previous point. It is the other way
around. Right now the spec dictates how to resolve (through employing a
view-specific `default-catalog` field). The proposal is suggesting to get
out of this space and let engines handle it similar to how they handle all
identifiers.

On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> > I thought "default-catalog" could be set via the USE command.
>
> Benny, I think this is a misconception or miscommunication. The USE
> command has no impact on the `default-catalog` field. In fact, the
> proposal's direction is exactly to establish that USE command should
> influence how tables are resolved, same like everywhere else. Right now it
> is not the case under the current spec.
>
>
> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <bt...@gmail.com> wrote:
>
>> > there is no SQL construct today to explicitly set default-catalog
>>
>> I thought "default-catalog" could be set via the USE command.
>>
>> I generally agree with Dan about requiring consistent catalog names.  I
>> think that's the lesser evil compared to Iceberg specifying how engines
>> should resolve identifiers.  Another thing to consider is that identifier
>> resolution can be very expensive at query validation time if identifiers
>> need to be looked up from a bunch of places.  Hopefully, it should be
>> possible to define a view in such a way that identifiers can be resolved on
>> the first try.
>>
>> Benny
>>
>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> Hi Rishabh,
>>>
>>> You're right that the proposal touches on two aspects, and resolution
>>> rules are one of them. The other aspect is the proposal's position that
>>> table identifiers should be stored in metadata exactly as they appear in
>>> the view text (e.g., even if they're two-part or partially qualified),
>>> along with their corresponding UUIDs for validation. This applies both to
>>> referenced input tables and the storage table identifier in materialized
>>> views.
>>>
>>> We may be able to converge on this storage format even if we haven't yet
>>> converged on the resolution fallback rules. I believe both resolution
>>> strategies currently being discussed would still lead to storing
>>> identifiers in this way.
>>>
>>> I'm supportive of moving forward with consensus on the identifier
>>> storage format. That said, we may continue to run into questions related to
>>> resolution during implementation. For example: Should the storage table
>>> identifier follow the same default-catalog and default-namespace resolution
>>> behavior as other table references?
>>>
>>> Thanks,
>>> Walaa.
>>>
>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia <
>>> bhatiarishabh63@gmail.com> wrote:
>>>
>>>> Hello Walaa,
>>>>
>>>> Thanks for starting this discussion.
>>>>
>>>> I think we should decouple at least the MV Spec from the proposal to
>>>> change the current behavior of view resolution.
>>>>
>>>> We can continue having the discussion if the current view spec needs to
>>>> be changed or not. Based on the decision at a later point if required we
>>>> can update the view resolution rule.
>>>>
>>>>
>>>> Thanks,
>>>> Rishabh
>>>>
>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> Correction of typo: both engines seem to set default-catalog to the
>>>>> view catalog if it is defined, or to null if the view catalog is not
>>>>> defined.
>>>>>
>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> Hi Dan,
>>>>>>
>>>>>> Thanks again for your response.
>>>>>>
>>>>>> I agree that catalog renaming is an environmental event, but it's a
>>>>>> real one that happens frequently in practice.
>>>>>> Saying that the Iceberg spec cannot accommodate something as common
>>>>>> as catalog renaming feels very restrictive, and could make the spec less
>>>>>> practical, even unusable, for real-world deployments.
>>>>>> I’m sharing this from the perspective of a large data lake
>>>>>> environment where views are heavily deployed and operationalized.
>>>>>>
>>>>>> Further, it's worth noting that the table spec is resilient to
>>>>>> catalog renaming, but the view spec is not. If we have an opportunity to
>>>>>> make the view spec similarly resilient, I wonder why not?
>>>>>> Both specifications are deterministic in their definition, but one is
>>>>>> more fragile to environmental changes than the other. Improving resilience
>>>>>> does not sacrifice determinism. It simply makes views safer and more
>>>>>> portable over time.
>>>>>>
>>>>>> Separately, given that there is no SQL construct today to explicitly
>>>>>> set default-catalog at creation time, what is the intuition behind how
>>>>>> engines like Spark and Trino currently assign default-catalog?
>>>>>> Today, both engines seem to set default-catalog to null if the view
>>>>>> catalog is defined, or to the view catalog if not.
>>>>>> What was the intended thought process behind this behavior?
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa
>>>>>>
>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dw...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Walaa,
>>>>>>>
>>>>>>> > tables inside views remain reachable after a catalog rename
>>>>>>>
>>>>>>> This problem stems from the exact environmental/configuration issue
>>>>>>> that we should not be trying to address.  I don't think we would expect
>>>>>>> references to survive a catalog rename.  That's not something covered by
>>>>>>> the spec and needs to be handled separately as a platform-level migration
>>>>>>> specific to the affected environment.
>>>>>>>
>>>>>>> The identifier resolution logic is clear and deterministic.  It
>>>>>>> should not matter whether an engine resolves and encodes the
>>>>>>> default-catalog or leaves it to the resolution rules.
>>>>>>>
>>>>>>> The issue isn't with how the spec is defined, but rather view
>>>>>>> behavior when you start altering the environment around it, which isn't
>>>>>>> something we should be trying to define here.
>>>>>>>
>>>>>>> -Dan
>>>>>>>
>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Dan,
>>>>>>>>
>>>>>>>> Thanks for chiming in.
>>>>>>>>
>>>>>>>> I believe the issues we’re seeing now go beyond just catalog naming
>>>>>>>> consistency. The behavior around default-catalog itself introduces
>>>>>>>> resolution inconsistencies even when catalog names are consistent.
>>>>>>>> For example:
>>>>>>>>
>>>>>>>> * When default-catalog is set to null, tables inside views remain
>>>>>>>> reachable after a catalog rename. But if it is set to a non-null value,
>>>>>>>> table references will break.
>>>>>>>>
>>>>>>>> * default-catalog causes table references inside views to be early
>>>>>>>> bound (i.e., bound at view creation time, especially when using a non-null
>>>>>>>> value), while table references inside standalone queries are late bound
>>>>>>>> (bound at query time). This creates inconsistencies when resolving the same
>>>>>>>> table name inside and outside views, even within the same job.
>>>>>>>>
>>>>>>>> * It causes Spark's and Trino behavior to drift from the spec.
>>>>>>>> There is no way to fully align Spark's behavior without making invasive
>>>>>>>> changes to the Spark SQL grammar and the View DataSource API (specifically
>>>>>>>> on the CREATE side). This challenge would extend to other engines too. Both
>>>>>>>> Spark and Trino set this field based on a heuristic in today's
>>>>>>>> implementation.
>>>>>>>>
>>>>>>>> * With view nesting (views depending on views), these
>>>>>>>> inconsistencies amplify further, forcing users and engines to reason about
>>>>>>>> catalog resolution at every level in the view tree.
>>>>>>>>
>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with that
>>>>>>>> model. Migrated Hive views will have to unfollow that spec.
>>>>>>>>
>>>>>>>> How would you suggest approaching the engine-level changes required
>>>>>>>> to support the current default-catalog field?
>>>>>>>> Also, do you believe the Spark and Trino communities would align
>>>>>>>> around having table resolution behave inconsistently between queries and
>>>>>>>> views, or inconsistency between Iceberg and other types of views?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Walaa
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <dw...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was
>>>>>>>>> introduced, but I think we need to step back and align on what we are
>>>>>>>>> really attempting to support in the spec.
>>>>>>>>>
>>>>>>>>> The issues we're discussing largely stem from using multiple
>>>>>>>>> engines with cross catalog references and configurations where catalog
>>>>>>>>> names are not aligned.  If we have multiple engines that all have the same
>>>>>>>>> catalog names/configurations, the current spec implementation is well
>>>>>>>>> defined for table resolution even across catalogs.  The 'default-catalog'
>>>>>>>>> (and namespace equivalent) was intended to address the resolution within
>>>>>>>>> the context of the sql text, not to address catalog/naming inconsistencies.
>>>>>>>>>
>>>>>>>>> I feel like we're trying to adapt the original intent to address
>>>>>>>>> the catalog naming/configuration and would argue that we shouldn't attempt
>>>>>>>>> to do that as part of the spec.  Inconsistently named catalogs are a
>>>>>>>>> reality, but we should consider that a configuration/environmental issue,
>>>>>>>>> not something to solve for in the spec.
>>>>>>>>>
>>>>>>>>> We should support and advocate for consistency in catalog naming
>>>>>>>>> and define the spec along those lines.  The fact is that with all of the
>>>>>>>>> recent work that's gone into making catalogs pluggable, it makes more sense
>>>>>>>>> to just register catalog configuration with consistent names (even if you
>>>>>>>>> have to duplicate the configuration for supporting existing
>>>>>>>>> readers/writers).  I think it's better to provide a path toward consistency
>>>>>>>>> than to normalize complicated schemes to workaround the issues caused by
>>>>>>>>> environmental/configuration inconsistencies.
>>>>>>>>>
>>>>>>>>> If the goal is to create clever ways to hack the late binding
>>>>>>>>> resolution to swap in different catalogs or make references contextual, I
>>>>>>>>> feel like that is something we should strongly discourage as it leads to
>>>>>>>>> confusion about what is resolved as part of the query.
>>>>>>>>>
>>>>>>>>> At this point, I don't see a good argument to add
>>>>>>>>> additional configuration or change the resolution behaviors.
>>>>>>>>>
>>>>>>>>> -Dan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul
>>>>>>>>> <ja...@mailbox.org.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> I think the intention with the "default-catalog" was that every
>>>>>>>>>> query engine uses it to store its session default catalog at the time of
>>>>>>>>>> creating the view. This way the view could be reused in another session.
>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the
>>>>>>>>>> default-catalog.
>>>>>>>>>>
>>>>>>>>>> Generally we have different environments we want to support with
>>>>>>>>>> the view spec:
>>>>>>>>>>
>>>>>>>>>> 1. Consistent catalog naming
>>>>>>>>>>
>>>>>>>>>> When the environment supports it, using consistent catalog names
>>>>>>>>>> can have a great benefit for multi-catalog, multi-engine setups. With
>>>>>>>>>> consistent catalog names, using the "default-catalog" field works without
>>>>>>>>>> any issues.
>>>>>>>>>>
>>>>>>>>>> 2. Inconsistent catalog naming
>>>>>>>>>>
>>>>>>>>>> This can be the case when different query engines refer to the
>>>>>>>>>> same physical catalog by different names. This often happens because
>>>>>>>>>> different query engines use different strategies to setup the catalogs. If
>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field does
>>>>>>>>>> not work because it is not guaranteed that the catalog name can be resolved
>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a better
>>>>>>>>>> solution for this use case, as it avoids catalog names altogether. It is
>>>>>>>>>> however limited to table references in the same catalog.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What do you think of introducing a view property that specifies
>>>>>>>>>> if the "default-catalog" or the "view catalog" should be used? This way,
>>>>>>>>>> you could use the "default-catalog" in environments where you can guarantee
>>>>>>>>>> consistent naming, but you would be able to directly fallback to the
>>>>>>>>>> "view-catalog" when you don't have consistent naming. The query engines
>>>>>>>>>> could set the default for this view property at creation time. Spark for
>>>>>>>>>> example could set it to automatically use the "view catalog".
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Jan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote:
>>>>>>>>>>
>>>>>>>>>> To help folks catch up on the latest discussions and
>>>>>>>>>> interpretation of the spec, I have summarized everything we discussed so
>>>>>>>>>> far at the top of the proposal document (here
>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
>>>>>>>>>> I have slightly updated the proposal to be in sync with the new
>>>>>>>>>> interpretation to avoid confusion. In summary:
>>>>>>>>>>
>>>>>>>>>> * Remove default-catalog and default-namespace fields from the
>>>>>>>>>> view spec completely.
>>>>>>>>>>
>>>>>>>>>> * Hence, we do not attempt to define separate view-level default
>>>>>>>>>> catalogs or namespaces.
>>>>>>>>>>
>>>>>>>>>> Instead:
>>>>>>>>>>
>>>>>>>>>> * If a table identifier inside a view lacks a catalog qualifier,
>>>>>>>>>> engines should resolve it using the current engine catalog at query time.
>>>>>>>>>>
>>>>>>>>>> * Reference table identifiers in the metadata exactly as they
>>>>>>>>>> appear in the view SQL text.
>>>>>>>>>>
>>>>>>>>>> * If an identifier lacks the catalog part at creation, it should
>>>>>>>>>> still lack a catalog in the stored metadata.
>>>>>>>>>>
>>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Walaa.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the
>>>>>>>>>>> fallback creates. Also just to be clear, at this point and after clarifying
>>>>>>>>>>> the current spec intentions, I am convinced that we should remove the
>>>>>>>>>>> default catalog and default namespace fields altogether.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Walaa.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I'd like to contribute my opinions on this:
>>>>>>>>>>>>
>>>>>>>>>>>> - I don't particularly like the current behavior of "default to
>>>>>>>>>>>> the view's catalog when default-catalog is not set".  Fundamentally, I
>>>>>>>>>>>> believe the intent of default-catalog and default-namespace is there to
>>>>>>>>>>>> help users write more concise SQL.
>>>>>>>>>>>> - spark session catalog is engine specific and I don't think we
>>>>>>>>>>>> should design something that says first use this catalog, then that
>>>>>>>>>>>> catalog.. or that catalog.  For example, resolving identifiers using
>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good.
>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no value
>>>>>>>>>>>> in putting views in the catalog to share with other engines
>>>>>>>>>>>> - Interoperability between different engine types is very hard
>>>>>>>>>>>> due to dialect issues... so I think we should focus on supporting different
>>>>>>>>>>>> clusters of the same engine type on a shared catalog.  For example, AI and
>>>>>>>>>>>> BI clusters on Spark sharing the same views in a REST catalog.
>>>>>>>>>>>>
>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the
>>>>>>>>>>>> lines of something Russell proposed last year:
>>>>>>>>>>>>
>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
>>>>>>>>>>>>
>>>>>>>>>>>> We've been looking at this interoperable identifier problem
>>>>>>>>>>>> through the lens of catalog resolution but maybe the right approach is
>>>>>>>>>>>> really about templating.
>>>>>>>>>>>>
>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a view to
>>>>>>>>>>>> span catalogs to support non-Iceberg tables.   Also, the default-catalog
>>>>>>>>>>>> property could be templated as well.
>>>>>>>>>>>>
>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>> Benny
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark
>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL extensions and/or
>>>>>>>>>>>>> Spark catalog APIs for that?
>>>>>>>>>>>>>
>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I shared
>>>>>>>>>>>>> regarding many resolution methods not consistently being followed in
>>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view
>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to a non-null
>>>>>>>>>>>>> value. If it helps, I can share concrete examples.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu <
>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The core issue is on the fall back behavior when
>>>>>>>>>>>>>> `default-catalog` is
>>>>>>>>>>>>>> not defined. Current view spec says the fallback should be
>>>>>>>>>>>>>> the catalog
>>>>>>>>>>>>>> where the view is defined. It doesn't really matter what the
>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>> is named (catalogX) by the read engine.
>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog, this is
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior.
>>>>>>>>>>>>>> - If a view refers to tables from another catalog, catalog
>>>>>>>>>>>>>> names
>>>>>>>>>>>>>> should be included in the reference name already. So no
>>>>>>>>>>>>>> ambiguity
>>>>>>>>>>>>>> there either.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate
>>>>>>>>>>>>>> problem, which
>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only
>>>>>>>>>>>>>> recommend that
>>>>>>>>>>>>>> catalog should be named consistently across usage for better
>>>>>>>>>>>>>> interoperability on name references.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This proposal is to change the fallback behavior to engine's
>>>>>>>>>>>>>> session
>>>>>>>>>>>>>> default catalog. I am not sure it is better than the current
>>>>>>>>>>>>>> fallback
>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this idea.
>>>>>>>>>>>>>> Spark resolves table identifiers during view creation using the session’s
>>>>>>>>>>>>>> default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for not
>>>>>>>>>>>>>> conforming
>>>>>>>>>>>>>> to the spec.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Hi Jan,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to
>>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of
>>>>>>>>>>>>>> default-catalog:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Here is the real catch:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view
>>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist before its
>>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible in Spark’s
>>>>>>>>>>>>>> implementation, where `default-catalog` is not used.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting would
>>>>>>>>>>>>>> require extending SQL syntax and engine APIs to promote it to a first-class
>>>>>>>>>>>>>> view concept. This would be intrusive, non-intuitive, and realistically
>>>>>>>>>>>>>> very difficult to standardize across engines.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this idea.
>>>>>>>>>>>>>> Spark resolves table identifiers during view creation using the session’s
>>>>>>>>>>>>>> default catalog, not a supplied `default-catalog`.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time
>>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding model between
>>>>>>>>>>>>>> tables vs views (early vs late), and between tables in views and in queries
>>>>>>>>>>>>>> (again early vs late). For example, views and tables in queries can
>>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when they are used
>>>>>>>>>>>>>> inside views -- it even applies to views inside views, which makes this
>>>>>>>>>>>>>> very hard to reason about considering nesting.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>> > Walaa
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> @Walaa:
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW statement
>>>>>>>>>>>>>> the query engine knowns which catalog the view is being created in. So even
>>>>>>>>>>>>>> though we typically use late binding to resolve the view catalog at query
>>>>>>>>>>>>>> time, it can also be used at creation time.
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> The query engine would need to keep track of the "view
>>>>>>>>>>>>>> catalog" where the view is going to be created in. It can use that catalog
>>>>>>>>>>>>>> to resolve partial table identifiers if "default-catalog" is not set.
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where partial
>>>>>>>>>>>>>> identifiers in the view query resolve to a different catalog compared to
>>>>>>>>>>>>>> using them outside of a view.
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT * from
>>>>>>>>>>>>>> sales.orders;
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>> "catalogA", which is the view's catalog.
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> Jan
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote:
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> For example, if we want to validate that the tables
>>>>>>>>>>>>>> referenced in the view exist, how can we do that when default-catalog isn't
>>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet?
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we
>>>>>>>>>>>>>> validate that a table exists without a default catalog, or do we always use
>>>>>>>>>>>>>> the current session catalog?
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> Thanks,
>>>>>>>>>>>>>> >> Manu
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> Hi Jan,
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> I think we still share the same understanding. Just to
>>>>>>>>>>>>>> clarify: when I referred to late binding as “similar” to the proposal, I
>>>>>>>>>>>>>> was acknowledging the distinction between view-level and table-level
>>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model.
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> That said, this still raises an interesting question and
>>>>>>>>>>>>>> a potential gap: if default-catalog is only defined at query time, how
>>>>>>>>>>>>>> should resolution work during view creation? For example, if we want to
>>>>>>>>>>>>>> validate that the tables referenced in the view exist, how can we do that
>>>>>>>>>>>>>> when default-catalog isn't defined, since the view hasn't been created or
>>>>>>>>>>>>>> loaded yet?
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> Thanks,
>>>>>>>>>>>>>> >>> Walaa.
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view catalog is
>>>>>>>>>>>>>> resolved at query time.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish
>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL statements. The
>>>>>>>>>>>>>> important part is that the physical catalog of the view and the tables
>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create a view in a
>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in your first
>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same physical
>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the behavior should
>>>>>>>>>>>>>> still work.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late binding
>>>>>>>>>>>>>> applies to the view catalog name and by extension to all partial table
>>>>>>>>>>>>>> references when no "default-catalog" is present. Resolving the view catalog
>>>>>>>>>>>>>> name at query time is not opposed to storing the view metadata in a catalog.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Thanks
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Jan
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Hi Jan,
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced in a
>>>>>>>>>>>>>> SQL statement. That statement contains the information for the query engine
>>>>>>>>>>>>>> to resolve the catalog of the view.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means:
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM
>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s catalog.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT * FROM
>>>>>>>>>>>>>> catalogB.namespace.view (after renaming catalogA to catalogB, and keeping
>>>>>>>>>>>>>> everything else the same), then catalogB becomes the view’s catalog.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to me
>>>>>>>>>>>>>> like the catalog is resolved at query time, based on how the view is
>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some sort of a
>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to using some
>>>>>>>>>>>>>> catalog that "stores" the view definition.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>> >>>> Walaa
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Hi Walaa,
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of non-determinism.
>>>>>>>>>>>>>> Let me try to address your questions.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec: The
>>>>>>>>>>>>>> view is executed when it's being referenced in a SQL statement. That
>>>>>>>>>>>>>> statement contains the information for the query engine to resolve the
>>>>>>>>>>>>>> catalog of the view. The query engine then uses that information to fetch
>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily keep track
>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then use that
>>>>>>>>>>>>>> information to resolve the table references in the views SQL definition in
>>>>>>>>>>>>>> case no default catalog is specified.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be
>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would assume the
>>>>>>>>>>>>>> view can be created in any catalog.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> I think your point is really valuable because the
>>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For example
>>>>>>>>>>>>>> for the following statement:
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS SELECT *
>>>>>>>>>>>>>> from sales.orders;
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA", the
>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just referencing
>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because without a
>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would default to
>>>>>>>>>>>>>> "catalogA".
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> However, I like the current design of the view spec,
>>>>>>>>>>>>>> because it has the "closure" property. Because of the fact that the "view
>>>>>>>>>>>>>> catalog" has to be known when executing a view, all the information
>>>>>>>>>>>>>> required to resolve the table identifiers is contained in the view metadata
>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier
>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders portability.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Jan
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Hi Jan,
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point before
>>>>>>>>>>>>>> going deeper:
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback
>>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table identifiers alone,
>>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed fallback to the
>>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g.,
>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects depending solely
>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So determinism isn’t
>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The only
>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the proposed use of
>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for correctness.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help
>>>>>>>>>>>>>> clarify two things in the context of the current spec:
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog” stored,
>>>>>>>>>>>>>> so that an engine knows to fall back to it if default-catalog is null?
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the session's
>>>>>>>>>>>>>> default catalog (i.e., without specifying a catalog) in the current Iceberg
>>>>>>>>>>>>>> spec?
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> These questions are important because if we can’t
>>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then defaulting to
>>>>>>>>>>>>>> it is problematic. And if views can't be created in the default catalog,
>>>>>>>>>>>>>> then the fallback rule doesn’t generalize.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>>>>>> >>>>> Walaa.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
>>>>>>>>>>>>>> <ja...@mailbox.org.invalid> <ja...@mailbox.org.invalid>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Hi Walaa,
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood
>>>>>>>>>>>>>> correctly, you proposal is composed of three parts:
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for
>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for
>>>>>>>>>>>>>> "default-namepace"
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for
>>>>>>>>>>>>>> "default-catalog"
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the current
>>>>>>>>>>>>>> session setup is in my opinion the definition of "non-determinism". You
>>>>>>>>>>>>>> could be running the same query-engine and catalog-setup on different days,
>>>>>>>>>>>>>> with different default session catalogs (which is rather common), and would
>>>>>>>>>>>>>> be getting different results.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view always
>>>>>>>>>>>>>> produces the same results. The current behavior has some rough edges in
>>>>>>>>>>>>>> very niche use cases but I think is solid for most uses cases.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for
>>>>>>>>>>>>>> "default-namespace"
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Similar to the above.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current
>>>>>>>>>>>>>> implementation already uses late binding.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup more
>>>>>>>>>>>>>> robust. Which is great. However, having UUID validation still requires us
>>>>>>>>>>>>>> to have a portable table identifier specification. Even if we have the
>>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply isn't an
>>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog interface is defined
>>>>>>>>>>>>>> in terms of table identifiers.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and
>>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can use the
>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can only be done
>>>>>>>>>>>>>> after we used some identifier. Which means there is no way of using UUIDs
>>>>>>>>>>>>>> without a functioning catalog/identifier setup.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for
>>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my opinion. And I
>>>>>>>>>>>>>> think the current spec does a good job for multi-engine table identifier
>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional hardening
>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Thanks
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Jan
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Thanks Renjie!
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving
>>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text with table
>>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use the catalog
>>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret or use.
>>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated catalog that
>>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In such
>>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in `my_catalog.namespace1.table1`)
>>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to underlying
>>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the catalog
>>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs actually behave
>>>>>>>>>>>>>> in practice.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Thanks,
>>>>>>>>>>>>>> >>>>>> Walaa.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu <
>>>>>>>>>>>>>> liurenjie2008@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> Hi, Walaa:
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal.
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have some
>>>>>>>>>>>>>> concerns with resolving catalog names on the fly with query engine defined
>>>>>>>>>>>>>> catalog names. This introduces some flexibility at first glance, but also
>>>>>>>>>>>>>> makes misconfiguration difficult to explain.
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store
>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may introduce
>>>>>>>>>>>>>> errors that's difficult to understand for user.
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Hi Everyone,
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and
>>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a vote next
>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The proposal
>>>>>>>>>>>>>> outlines a strategy for handling table identifiers in Iceberg view
>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, and engine
>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read time (late
>>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based validation to
>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It also revises
>>>>>>>>>>>>>> how default-catalog and default-namespace are handled (defaulting both to
>>>>>>>>>>>>>> the session context if not explicitly set) to better align with engine
>>>>>>>>>>>>>> behavior and improve cross-engine interoperability.
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts.
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the
>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog names
>>>>>>>>>>>>>> in the spec refer to logical catalogs—i.e., the first part of a three-part
>>>>>>>>>>>>>> identifier. These correspond to Spark's DataSourceV2 catalogs, Trino
>>>>>>>>>>>>>> connectors, and similar constructs. This is a level of abstraction above
>>>>>>>>>>>>>> physical catalogs, which are not referenced or used in the view spec. The
>>>>>>>>>>>>>> reason is that table identifiers in the view definition/text itself refer
>>>>>>>>>>>>>> to logical catalogs, not physical ones (since they interface directly with
>>>>>>>>>>>>>> the engine and not a specific metastore).
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>> Thanks,
>>>>>>>>>>>>>> >>>>>>>>> Walaa.
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun <
>>>>>>>>>>>>>> sungwyun8@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think view
>>>>>>>>>>>>>> portability is a very important topic for us to continue discussing as it
>>>>>>>>>>>>>> relies on many assumptions within the data ecosystem for it to function
>>>>>>>>>>>>>> like you've highlighted well in the document.
>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may
>>>>>>>>>>>>>> impact the permission questions the engines will be asking, and whether
>>>>>>>>>>>>>> that is the desired behavior.
>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>> Sung
>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard Tudenhöfner
>>>>>>>>>>>>>> <et...@apache.org> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've
>>>>>>>>>>>>>> added a few comments to get a better understanding of how this will look
>>>>>>>>>>>>>> like in the actual implementation.
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>> Eduard
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin
>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our discussion on
>>>>>>>>>>>>>> how to reference table identifiers from Iceberg metadata, a key aspect of
>>>>>>>>>>>>>> the view specification, particularly in relation to the MV (materialized
>>>>>>>>>>>>>> view) extensions.
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a few
>>>>>>>>>>>>>> community members to better understand how the current spec is being
>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new proposal on how
>>>>>>>>>>>>>> table identifier references could be represented in metadata.
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look
>>>>>>>>>>>>>> forward to your feedback and working together to move this forward so we
>>>>>>>>>>>>>> can finalize the MV spec as well.
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>