You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "JanKaul (via GitHub)" <gi...@apache.org> on 2024/08/29 12:25:29 UTC
[PR] Materialized View Spec [iceberg]
JanKaul opened a new pull request, #11041:
URL: https://github.com/apache/iceberg/pull/11041
This PR implements the Iceberg Materialized View Proposal #10043 by adding a section for Materialized Views to the View spec. It follows the design of the [proposal document](https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A).
The idea is to resolve any remaining questions before starting the voting process on the dev list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1737474791
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
+
+For that the additional field "refresh-state" is introduced as an opaque record in the table snapshot summary.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string. |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view. It has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#soure-table) records |
+| _required_ | `source-view-states` | A list of [source view](#soure-view) records |
Review Comment:
sounds good. I am good with separated lists too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1868916881
##########
format/view-spec.md:
##########
@@ -82,9 +98,13 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [partial identifier](#partial-identifier) of the storage table |
Review Comment:
I think there are two parts to this issue: Resolving the namespace and resolving the catalog.
I would always resolve the namespace of the storage table as I think this should provide the most consistency. Requiring the query engine to resolve a partial reference on every access seems more fragile to me. Another point is that the storage table identifier is set by the query engine which has the complete information about the identifier.
Resolving the catalog is more complicated as the discussion about the "catalog alias" problem shows. It makes sense to use an optional "catalog_name" here. The question is what happens if the value is not set. We have these options:
1. Use the same catalog for the storage table as for the view
2. Use the `default-catalog` from the view version (if I understand you correctly Dan)
I'm in favor of Option 1 because my understanding is that the `default-catalog` relates to the references in the view representation and not to the view itself. I find it natural that the storage table is in the same catalog as the view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1910988506
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
Review Comment:
(also update all other `full identifier` references to `table identifier`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1851174083
##########
format/view-spec.md:
##########
@@ -42,12 +42,25 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
Review Comment:
I think its a bit long/detailed, if this is more of an intro, high level paragraph, how about just
```Materialized views maintain additional metadata to track its storage table and its freshness.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2086105999
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
Review Comment:
I think a link is a great idea but I also agree that I would duplicate the information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2237244518
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table and refresh metadata.
+Refresh metadata contains information about the "source tables", tables referenced in the query definition of the materialized view.
+The storage table can be "fresh", "stale" or "invalid", depending on the following the following situations:
Review Comment:
want to clarify that the state is a computed value during read and is not sth stored in the refresh state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1963947652
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
Review Comment:
I remember Dan saying that he would like to keep the Table spec clear of any MV structures. And I think it's good that all MV related information can be found in one place.
But I don't have a strong opinion on this and we could move it to the table spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2288776869
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table which contains additional refresh metadata.
Review Comment:
Maybe it is my interpretation problem. `which` can be interpreted as the `pointer` or the `storage table`. I know it meant the later. The following paraphrase can avoid the potential misinterpretation.
```
Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
```
I also think the sentence below may not be needed here. It is not clear from the description. maybe better just leave it out and let the later refresh-state section coveres it.
```
Refresh metadata contains information about the "source tables" and/or "source views", tables referenced in the query definition of the materialized view.
```
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table which contains additional refresh metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", tables referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following the following situations:
Review Comment:
duplicate `the following` in this line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "sfc-gh-ibelianski (via GitHub)" <gi...@apache.org>.
sfc-gh-ibelianski commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2593702889
##########
format/view-spec.md:
##########
@@ -160,6 +181,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
Review Comment:
in this case (Storage Table Identifier). this is always a table , and nvere a view . Right?
```suggestion
| _required_ | `name` | A string specifying the name of the table |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "yyanyy (via GitHub)" <gi...@apache.org>.
yyanyy commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2412249832
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of the source tables.
+* **stale** -- The `snapshot_id`s do not match, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
I personally don't have first hand experience working with MV by here's my 2 cents: I'm not completely sure if we need to couple this with the lineage discussion, since I feel that they may serve different purposes. I think from the doc Steven shared, the main purpose and advantage of option 2 is to help engine to determine if the MV is stale or not. Obviously the exact criteria for determining this is engine specific, but just from a high level guess, I think when the use case of someone creating MV1 from MV2 and MV3 emerges, most likely this user/engine would expect MV1 to be refreshed based on MV2 and MV3's data most of the time, instead of recursively obtain the most deeply nested source table and start from there; otherwise it feels that there's not much point creating MV2 and MV3 as materialized view and use them as the children for MV1, normal views would be enough. Because of this, I think the fact that it's a nested MV makes the "materialized" part of the child views more
interesting for engine processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2795141897
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
may be can give example of the life of MV" ? specifically :
is it ok to have MV without table ( seem based on the spec the answer is NO
it also means there is no easy way to support scenario where MV is created (published)
refresh is separate later transactions .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2801517081
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
In my mind, it seems natural for the `CREATE MATERIALIZED VIEW` DDL to create the view and storage table objects together in the catalog. Just that initially, the storage table will be empty until refresh kicks in. One benefit of this choice is to apply the storage configs (like partition, sort order etc.) from DDL to the storage table. Otherwise, those storage configs would need to be persisted to the view object when executing the DDL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2806471933
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
Is initially empty storage table would be easily distinguishable from recently refreshed but (happened to be) empty (as query retuned nothing) storage tables ? IMO:
1) the answer must be YES (these are materially different scenarios)
2) spec needs be very crisp / explicit about what that preinitialized state looks like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2791149923
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and nested materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of nested materialized views |
+| `view` | An Iceberg view, including nested materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
Review Comment:
just clarifying the entries views and tables for materialized views are impossible to match ( without reading all of the view definitions right . I am wondering if there are reasonable scenarios where having this map would be helpful? )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3011236572
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
Review Comment:
> represents the logical query definition of the view.
I don't think this is quite accurate. Usually 'freshness' is related to the definition and data, not just the definition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3011229162
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
If we include the "catalog" name, we get the issue that different query engines might resolve it differently.
That's why we restrict it to the same catalog to avoid the issue. And it covers the main use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3011325711
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
+| _optional_ | `ref` | Branch name of the source table being referenced in the view query |
+
+When `ref` is `null` or not set, it defaults to "main".
+
+#### Source view state
+
+A source view record captures the state of a source view at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `view` |
+| _required_ | `name` | A string specifying the name of the source view |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
+| _required_ | `uuid` | The uuid of the source view |
+| _required_ | `version-id` | The version-id of the source view that was read during the refresh operation |
+
+#### Storage table creation and configuration
+
+When processing a `CREATE MATERIALIZED VIEW` statement, query engines must:
+
+1. Create the storage table as a regular Iceberg table with any specified configurations (partitioning, sort order, compression, etc.).
Review Comment:
I don't feel lke we need to include this in the spec. The configurations set on the table are a implementation detail and don't need to be enforced by the spec. Some engines may allow setting these configurations, while others may not (and rely on the backing catalog implementation).
This is also not a query engine requirement as the catalog implementation should handle this (not the engine directly).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3014787371
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
Review Comment:
@igorbelianski-cyber @stevenzwu @bennychow What do you think about including all distinct source states in case of the diamond pattern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242285387
##########
format/view-spec.md:
##########
@@ -190,92 +190,93 @@ The table identifier for the storage table that stores the precomputed results.
### Storage table metadata
This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
-The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
| Requirement | Field name | Description |
|-------------|-----------------|-------------|
| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
#### Freshness
-A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
-Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
Review Comment:
s/transitively/transitive/
intermediate is not a precise term here i think it should directly referenced ( or referenced through views) or similar phrasing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242285387
##########
format/view-spec.md:
##########
@@ -190,92 +190,93 @@ The table identifier for the storage table that stores the precomputed results.
### Storage table metadata
This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
-The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
| Requirement | Field name | Description |
|-------------|-----------------|-------------|
| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
#### Freshness
-A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
-Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
Review Comment:
s/transitively/transitive/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2875579143
##########
format/view-spec.md:
##########
@@ -322,3 +453,96 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```
Review Comment:
This improves the readability of the generated page:
```suggestion
```json
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-3988164247
> Is there a reference implementation draft PR corresponding to the MV spec?
@findinpath There is WIP PR https://github.com/apache/iceberg/pull/15000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2882375723
##########
format/view-spec.md:
##########
@@ -322,3 +453,96 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
Review Comment:
I would like to avoid cluttering the spec document. I think these scenarios are better suited as a test like the ones here: [https://github.com/apache/iceberg/tree/main/core/src/test/resources](https://github.com/apache/iceberg/tree/main/core/src/test/resources)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242318057
##########
format/view-spec.md:
##########
@@ -190,92 +190,93 @@ The table identifier for the storage table that stores the precomputed results.
### Storage table metadata
This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
-The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
| Requirement | Field name | Description |
|-------------|-----------------|-------------|
| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
#### Freshness
-A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
-Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
-**Consumer behavior:**
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
-When evaluating freshness, consumers:
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
-- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
-- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
-- May parse the view definition to implement more sophisticated policies.
-- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
-- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+##### Producer flexibility
-**Producer behavior:**
-
-Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
-Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
-Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
When writing the refresh state, producers:
-- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
-- If the source state cannot be determined for all objects (for example, for non-Iceberg tables or non-deterministic functions) may leave the source states list empty.
-- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
Review Comment:
Must ....... they chose to track.
sounds awkward and didn't we discuss a case of producers being able to track more things than consumers. And the list is meant for consumer.
I think we can just use "Should" here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242560284
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
I agree that generally the MV and storage table will live in the same catalog, but I don't think we need to force that as a requirement. Just change the 'MUST' to a 'SHOULD' and we leave enough room for implementations to differ. The main point is that the storage table needs to be addressable (not that it live in the same catalog).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2791952442
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and nested materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of nested materialized views |
Review Comment:
What can be included in the list is explained in the "producer behavior" section at line 215. Do you think I should repeat the information here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3033123038
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
Review Comment:
I'd like to propose some simplificaition by decomposing this aspect to _three independent dimensions._ Specifically around:
* What constitutes sufficient state (logically)
* What flexibility producers have
* What options consumers have
Finally, I would connect the above with how child table identifier resolution works across the view/table boundary.
### 1. What is logically a sufficient refresh state
The definition of "sufficient" depends on how intermediate materialized views are treated:
**Case A: No intermediate MVs (all sources are base tables, optionally through intermediate views)**
Sufficient state = snapshot IDs of all leaf tables referenced by the view's query, in addition to intermediate view version IDs. This is the straightforward case.
**Case B: Intermediate MVs treated as views (transparent expansion)**
The engine expands intermediate MVs into their underlying queries. This reduces to Case A — sufficient state is the snapshot IDs of all deep leaf tables, plus version IDs of all intermediate views traversed during expansion.
**Case C: Intermediate MVs treated as tables (opaque boundaries)**
The engine treats intermediate MVs as materialized data sources. Sufficient state = snapshot IDs of the intermediate MVs' storage tables. No expansion beyond the MV boundary. Freshness of the intermediate MV is that MV's own concern.
Without guidelines on what consitutes sufficiency, consumers have no way to distinguish "the producer intentionally recorded partial state" from "the producer recorded complete state." A consumer seeing two source entries has no signal whether that covers all sources or just the ones the producer chose to track. These definitions can be added to the spec for reference.
### 2. Producer options
Producers decide what state to record at refresh time. This can be partial — a producer may only track sources from the same catalog, may skip non-Iceberg sources, or may not expand intermediate MVs.
### 3. Consumer options
Consumers have two strategies:
- **Trust the recorded state.** Simple to implement, but only correct if the producer recorded complete state. No way to verify this from the refresh state alone.
- **Parse the view query independently.** The consumer re-analyzes the SQL, identifies all source tables, and checks whether the recorded state covers them. Correct but expensive — requires SQL parsing and cross engine dialect translation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3320636746
##########
format/view-spec.md:
##########
@@ -322,3 +449,215 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+- `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+- `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+- `C` (regular view or materialized view, varies by strategy): `SELECT ... FROM F JOIN G ON ...`
+- `D` (regular view or materialized view, varies by strategy): `SELECT ... FROM H WHERE ...`
+- `E`, `F`, `G`, `H`: base Iceberg tables
+
+### Strategy 1: Track all nested tables and views (no nested MVs)
+
+The view query reads only base tables and regular views. The refresh state tracks snapshot IDs of all deeply nested base tables and version IDs of all views traversed. Reuse of the storage table is sensitive to changes in any of them.
+
+`C` and `D` are regular views.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [VIEW] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [VIEW] <-- recorded in A: version-id: 7
+ ├── F [TABLE] <-- recorded in A: snapshot-id: 102
+ └── G [TABLE] <-- recorded in A: snapshot-id: 103
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query reads from materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is **not** recorded.
+
+`C` and `D` are materialized views, treated as views (expanded).
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV — expanded as view] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [MV — expanded as view] <-- recorded in A: version-id: 7
Review Comment:
This example would be more interesting if the storage table for C was stale so that this example didn't have the same refresh state as Strategy 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3318972836
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies in the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an upstream materialized view's storage table) and/or a `view` entry. Upstream materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track (diamond dependency pattern).
Review Comment:
(diamond dependency pattern). reads like a TODO that wasn't addressed was it meant to be here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3465274748
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+A snapshot whose refresh state violates a `Must` rule is invalid; consumers may treat it as if it had no `refresh-state`.
+
+##### Consumer options
+
+Consumers may use any combination of the following to assess the storage table:
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
Review Comment:
I agree with Igor that we should generally try to reuse existing terms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4050453756
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
Review Comment:
Missing subject. **consumer** can fail.
##########
format/view-spec.md:
##########
@@ -322,3 +454,96 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
Review Comment:
Since catalog will be required because we only want to capture fully resolved identifiers, we should add "catalog: prod" here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4143943053
##########
format/view-spec.md:
##########
@@ -160,6 +178,108 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view's precomputed data becomes stale as the tables and views referenced in its query definition change over time. Freshness determines whether the precomputed data accurately represents the logical query definition at the current state of its dependencies.
+
+Different systems define freshness differently, based on how much of the dependency graph must be current. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, "fresh" can mean strict end-to-end consistency, acceptable lag, or policy/version compliance.
+
+A materialized view is considered fresh when its precomputed data meets the freshness criteria defined by the consumer's evaluation policy. When these criteria are not met, the materialized view is considered stale.
+
+#### Refresh state
+
+The refresh state record captures the unique dependencies in the materialized view's dependency graph. These dependencies include source Iceberg tables, views, and nested materialized views that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+
+**Consumer evaluation:**
+- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`. A materialized view is fresh if `refresh-start-timestamp-ms` is within the window `[now - max-staleness-ms, now]`.
+- The consumer may additionally compare the `source-states` list against the states loaded from the catalog. If this evaluation determines the materialized view is fresh, it overrides the coarse-grained evaluation result.
+- The consumer may parse the view definition to implement a more sophisticated policy.
+- When a materialized view is considered stale, the consumer can fail, refresh inline, or treat the materialized view as a logical view. The consumer must not consume from the storage table when the materialized view is stale.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Materialized views can reference source objects of different types, such as Iceberg tables, view, and materialized views. Source state records have a common field `type` that determines the form, which can be one of the following:
+
+* `table`: An Iceberg table
+* `view`: An Iceberg view
+* `materialized-view`: An Iceberg materialized view
+
+The metadata fields for each type are defined below:
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
Review Comment:
I don't have a lot of experience with this, so take this with a grain of salt. I'm wondering whether the `sequence-number` or something similar can help to recover from missing snapshots. I was under the impression that you actually need the exact state at a certain snapshot to be able to refresh incrementally.
Let's say the snapshot of the recorded `snapshot-id` in `source-table-states` has been removed, but I have a snapshot with a close `sequence-number` (either before or after). I don't see how this will allow me to accurately pick up the incremental computation for the storage table. The content of the storage table is closely tied to the state recorded at the particular snapshot-ids in the `source-table-states`. If the state of the source tables is different, even just a small sequence-number difference, it still means that the content of the storage table can be very different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "osscm (via GitHub)" <gi...@apache.org>.
osscm commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4126794820
##########
format/view-spec.md:
##########
@@ -63,11 +75,13 @@ The view version metadata file has the following fields:
| _required_ | `versions` | A list of known [versions](#versions) of the view [1] |
| _required_ | `version-log` | A list of [version log](#version-log) entries with the timestamp and `version-id` for every change to `current-version-id` |
| _optional_ | `properties` | A string to string map of view properties [2] |
+| _optional_ | `max-staleness-ms` | The maximum time interval in milliseconds during which changed source table snapshots are considered fresh enough to skip refreshing [3] |
Review Comment:
Curious if this survived into the current draft — I don't see a `max-staleness-ms` field (or any staleness-window config) anywhere in the current spec text, and the "Recency policy" bullet later in the doc still says "falls within a staleness window" without anything defining what that window actually is. Not sure if that's leftover wording from a since-superseded revision, or an intentional decision to leave the staleness bound out of the interchange metadata entirely and let each engine define its own out-of-band policy.
On the Option 1 vs. Option 2 debate (refresh-start-timestamp vs. earliest-changed-source-snapshot-timestamp): for what it's worth, StarRocks' `mv_rewrite_staleness_second` property already ships something closer to Option 2/delayed-view semantics — it computes staleness as the gap between the base tables' current data timestamp and the point at which the MV's freshness was last *confirmed* by a complete refresh, and allows query rewrite to use the MV as long as that gap stays within the configured bound, rather than gating purely on "how long ago did the last refresh start." One detail from that implementation that might be worth folding in if this comes back into the spec: it has an explicit guard for when a base table's timestamp *regresses* (e.g. after `rollback_to_snapshot`, or dropped newest partitions) — the staleness window is deliberately not trusted in that case, since a naive time-based check could otherwise serve rows that no longer exist upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4098332333
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
Review Comment:
```suggestion
A materialized view is **fresh** when the storage table represents the result of the current view query. Freshness does not dictate serving policy: each engine defines its own policy for when to recompute the query and when to reuse the storage table.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4098332333
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
Review Comment:
```suggestion
A materialized view is **fresh** when the storage table represents the result of the current view query. Freshness does not dictate serving: each engine defines its own policy for when to recompute the query and when to reuse the storage table.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4094171037
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple path in the dependency graph.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its view metadata and one ore more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
Review Comment:
I've made it optional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1779799446
##########
format/view-spec.md:
##########
@@ -81,9 +93,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [full identifier](#full-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When 'storage-table' is `null` or not set, the entity is a common view, otherwise it is a materialized view.
+
Review Comment:
Given that its up to the engine to use it or not, I'm OK if we don't include this property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2469334725
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1742350128
##########
format/view-spec.md:
##########
@@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
+The storage table can have the states "fresh", "stale" or "invalid".
Review Comment:
maybe a minor change from `can have the states` to `can be in the state of` would avoid the misinterpretation. `have` can imply `containing the state field`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1744837644
##########
format/view-spec.md:
##########
@@ -81,9 +93,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [full identifier](#full-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When 'storage-table' is `null` or not set, the entity is a common view, otherwise it is a materialized view.
+
Review Comment:
We also discussed adding a view property:
**materialization.max-staleness-ms**
This is a hint to the materialization consumer about the grace period that a possibly stale materialization can still be used. This grace period may start when the refresh is triggered or when a source table or view has changed depending on implementation.
Do you think we should still have this property? It gets complicated when you have Iceberg views that reference non-Iceberg tables.
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
Review Comment:
Should we include **refresh-start-timestamp-ms** which is when the refresh is triggered and an approximation of the refresh state?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "chenjian2664 (via GitHub)" <gi...@apache.org>.
chenjian2664 commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2081144649
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of the source tables.
+* **stale** -- The `snapshot_id`s do not match, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
Should we consider the case where the current materialized view references another materialized view? If we determine that the current materialized view is not stale, we should also ensure that any referenced materialized views are not marked as invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1910784199
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view that precompute the data from the view query.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
Review Comment:
```suggestion
The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1851160036
##########
format/view-spec.md:
##########
@@ -158,6 +175,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Partial identifier
+
+The partial identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
Review Comment:
I would simplify this. ( no need to state 'additional metadata' or 'introduced' as its obvious from the context)?
```A snapshot property called 'refresh state' is set on the storage table to determine the freshness of the precomputed data in that table. ```
Can we also link to snapshot property section of the table spec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1855985153
##########
format/view-spec.md:
##########
@@ -42,12 +42,27 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view that precompute the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and information about the refresh state to determine if the data is still fresh.
+The refresh state is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
Review Comment:
Thanks a lot for the feedback.
The refresh state is actually something else (see line 200). But I've rephrased the section to hopefully avoid that confusion. Additionally, I created bullet points, as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #11041: Materialized View Spec
URL: https://github.com/apache/iceberg/pull/11041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1861267203
##########
format/view-spec.md:
##########
@@ -82,9 +98,13 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [partial identifier](#partial-identifier) of the storage table |
Review Comment:
I don't know if we should call it `partial`. the storage table identifier can be full and include catalog name.
##########
format/view-spec.md:
##########
@@ -82,9 +98,13 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [partial identifier](#partial-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When 'storage-table' is `null` or not set, the entity is a common view, otherwise it is a materialized view.
+The catalog of the storage table must be same as the one of the materialized view.
Review Comment:
I thought the consensus is either the catalog is the same as the one of the MV (implicit), or the catalog name can be specified explicitly too.
portability across engines is assumed with consistent catalog naming across engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "devozerov (via GitHub)" <gi...@apache.org>.
devozerov commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2497150365
When materialized view is created, two entities would be added to the catalog - a view and a storage table. From the engine perspective it is important to expose it as a single object during listing. Are there any rules how catalog implementors should deal with these objects? E.g., shall we expose the view via `listViews`, but filter-out the storage table for `listTables`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-3873040325
@InvisibleProgrammer looks like we missed your earlier questions/comments
> There is a field in view metadata called format-version. The spec says it should be 1. Can the format version be changed with this specification?
I believe you are talking about the view format version in the view spec. Since we are not introducing breaking changes to the view spec. We don't need to bump up the view spec version.
> was thinking about the storage table: at this point if we pick up a table, we just don't know if it is a storage table of a materialized view or just a regular table. Can the table metadata get a reference to the materialized view in case it is a storage table of a view? It would help other engines to protect the storage table from accidental table operations: if the engine is more strict, it can completely hides that table or disallow to do DML operations on that. If the engine less strict it can completely ignore that field.
MV storage tables have the `refresh-state` to distinguish with regular tables.
Yes, engines are recommended to protect the storage tables from direct DMLs and only allow MV refresh operations. Catalogs (especially REST) also plays an important role here. E.g., REST catalog may return the view and storage table metadata in one response.
> And lastly please let me bring up a question about the timeline
We made a lot progress, compromises, and consensus in the last a few months. I believe we have reached consensus in the current spec/design discussion for at least the initial phase. Agree with you that we can still evolve and improve the spec in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2809831576
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
> Otherwise you won't be able to store configuration options like partitioning, sorting, ..
The storage table configuration isn't the source of truth for the MV configuration. This configuration is engine specific and an engine may use view properties or something completely external.
> seems natural for the CREATE MATERIALIZED VIEW DDL to create the view and storage table objects together in the catalog.
I don't think the spec should require creating the objects together. Imagine this sequence of events:
1. User creates logical view version 1
2. User builds materialization and adds storage table to view version 2
3. User changes view SQL and view schema, removes storage table and commits view version 3
4. User builds new materialization and adds new storage table to view version 4
Basically, I don't see a need to couple the view with the same storage table.
Another reason to not couple:
For full refresh, the producer can decide whether its more efficient to truncate and rebuild the existing storage table or to create a new storage table and commit that with a new view version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3014525620
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
Review Comment:
With regards to freshness, to argument was as follows:
- all source tables need to be up-to-date for a storage table to be fresh
- a single out-of-date source table renders the storage table stale
- if any source table snapshot is out-of-date, the oldest snapshot is also out-of-date
- if the oldest snapshot is up-to-date, all other snapshots are up-to-date
- the oldest snapshot is enough information to determine freshness in the diamond pattern
Your comment is really relevant for incremental processing though. As the information what data has been processed is vital. I think we should then include all source table states, even different snapshots of the same table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3033444774
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
Review Comment:
In most cases, it is probably reasonable for the MV producer to determine the freshness intent. Sophisticated should have the capability to parse SQL and can implement different strategies.
I am not sure about those boolean flags that @bennychow mentioned that add unnecessary complexities to the spec.
What is the typical runtime latency for SQL syntax parsing? A few millis? Is it really important to optimize for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3033372305
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
Review Comment:
+1 to additional metadata to help the consumer with more determinism @bennychow.
By the way the first section `1. What is logically a sufficient refresh state` is just for reference and not for enforcement. I think we should state it to eliminate ambiguity. Sections 2 and 3 still give the flexibility we discussed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2380871796
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of the source tables.
+* **stale** -- The `snapshot_id`s do not match, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
@wmoustafa I think we discussed that storing the lineage inside the view was unnecessary and that having the refresh state in the storage table was enough. The engine that queries the view will have to parse, validate and check staleness in order to replace the view with the storage table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2381026317
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of all the source tables.
+* **stale** -- The `snapshot_id`s do not match for at-least one source table, indicating that a refresh operation needs to be performed to capture the latest source table changes.
+* **invalid** -- The current `version_id` of the materialized view does not match the `refresh-version-id` of the refresh state.
Review Comment:
I do not see `refresh-version-id` referenced elsewhere in the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2590258037
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of all the source tables, OR all source table snapshots that differ from the last refresh have timestamps within a configured staleness window.
+* **stale** -- The `snapshot_id`s do not match for at least one source table and at least one differing snapshot has a timestamp outside the configured staleness window, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
I still think it might be better to move the status part to a separate section in the end (like after the refresh state section). Would be better to describe the refresh state and the max staleness config first before talking about how to determine fresh/stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2587929339
##########
format/view-spec.md:
##########
@@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
I've added it as an additional field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2590206054
##########
format/view-spec.md:
##########
@@ -160,6 +181,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables, views, and materialized views in the materialized view's fully expanded query tree at refresh time. Source table states are stored in `source-table-states` and source view states in `source-view-states`. For source views, `source-view-states` includes indirect references — tables or views nested within other views (exluding MVs) but not directly referenced in the query.
+For source materialized views, both the source view and its storage table are included in the refresh state. Indirect references are excluded for materialized view sources; during read time, query engines may recursively expand the query tree to determine freshness. The refresh state has the following fields:
Review Comment:
we need a blank line before this line for this to become a separate paragraph.
We probably need some tweaks on the wording.
------------------------------
For directly referenced source materialized views, both the source view and its storage table are included in the refresh state. Indirect references from source materialized views are excluded in the refresh-state. During read time, a query engine recursively expand the query tree to determine freshness if it chooses to enforce recursive evaluation semantic.
The refresh state has the following fields:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3319488917
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies in the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an upstream materialized view's storage table) and/or a `view` entry. Upstream materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track (diamond dependency pattern).
Review Comment:
That if a table is referenced through multiple dependency paths, like in the diamond pattern, all of its distinct source states have to be recorded. In case different snapshots are referenced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3319895788
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query.
Review Comment:
What do you think about:
However, consumers may still decide to consume from a stale storage table based on their own policies.
This way we don't use refresh-state before it's defined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242765323
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+A snapshot whose refresh state violates a `Must` rule is invalid; consumers may treat it as if it had no `refresh-state`.
+
+##### Consumer options
+
+Consumers may use any combination of the following to assess the storage table:
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
Review Comment:
```suggestion
- **Refresh Staleness.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. Staleness bounds data age but does not establish freshness.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242354319
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
Review Comment:
this could also be the case if the MV is considered always fresh/consumer like the Postgres semantics. should we spell that out too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242591380
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+A snapshot whose refresh state violates a `Must` rule is invalid; consumers may treat it as if it had no `refresh-state`.
+
+##### Consumer options
+
+Consumers may use any combination of the following to assess the storage table:
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for intermediate materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table; otherwise it evaluates the view query in place of the storage table.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph. Each dependency is recorded in `source-states` as either a `table` entry (a base table or an intermediate materialized view's storage table) or a `view` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a base table in the dependency graph, or the storage table of an intermediate materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+An intermediate materialized view must be recorded as a single `table` entry referencing its storage table; recording it as a `view` entry is not permitted. The intermediate materialized view's own dependencies are reached recursively through its `refresh-state`.
Review Comment:
> An intermediate materialized view must be recorded as a single `table` entry referencing its storage table
`must` could be interpreted that a nested MV must be recorded in the source-state, which contradicts with earlier statement that producers can choose any subset.
> recording it as a `view` entry is not permitted
A nested MV should also have a view entry, right? if the nested MV's view definition changed, the source state can be used to detect the version mismatch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3254794742
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
At some point we made the decision to restrict it to the same catalog to avoid the catalog alias problem. Consequently the metadata currently doesn't have a "catalog" field. So currently you can't specify a different catalog. So in order to allows for a "SHOULD", we have to also change the metadata.
I think restricting it to the same catalog is a reasonable restriction that makes the usage simpler in many cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3397278429
##########
format/view-spec.md:
##########
@@ -42,12 +42,27 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables", "source views", and/or "source materialized views", which are the tables/views/materialized views referenced in the query definition of the materialized view.
+
## Specification
### Terms
* **Schema** -- Names and types of fields in a view.
* **Version** -- The state of a view at some point in time.
+* **Storage table** -- Iceberg table that stores the precomputed data of a materialized view.
+* **Refresh state** -- A record stored in the storage table's snapshot summary that captures the state of source tables and views at the time of the last refresh operation.
+* **Dependency graph** -- The graph of all source tables, views, and materialized views that a materialized view depends on, including nested dependencies.
+* **Source table** -- A table reference that occurs in the query definition of a materialized view.
Review Comment:
we define the sources as direct children here. In the refresh state, the field name is `source-states`. If we put those two together, it conflicts with the wording of `Producers may selectively choose a subset of their dependencies to record`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4082685633
##########
format/view-spec.md:
##########
@@ -322,3 +453,201 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
Review Comment:
In practice this will depend on the query engine what kind of pattern it uses to name storage tables. Typically you want to choose something that doesn't collide with the users table names, so in this example `__`(double underscore) was chosen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4082818578
##########
format/view-spec.md:
##########
@@ -160,7 +179,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+* **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+* **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple path in the dependency graph.
+* **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+* **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+* **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+* **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its view metadata and one ore more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `catalog` | A string specifying the name of the catalog. |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
+| _optional_ | `ref` | Branch name of the source table being referenced in the view query |
+
+When `ref` is `null` or not set, it defaults to `main`.
Review Comment:
`FOR VERSION AS OF` allows both a branch name as well as a snapshot identifier.
The current spec seems to omit the use case of referencing a table by a specific snapshot.
Given that we know about this use case / missing coverage in the spec, I think we should handle it to have the spec complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3967960238
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
Review Comment:
Iceberg snapshots internally contain the `schema-id` that was active at the time the snapshot was created. Therefore, to track schema evolution, engines do not need extra schema information injected into the refresh-state. During freshness evaluation, the query engine simply:
- Looks at the source table snapshot-id recorded in the MV's refresh-state.
- Checks the schema-id tied to that specific snapshot in the source table.
- Compares it against the source table's current schema-id.
If the schema IDs differ (e.g., a column was renamed or a default value was added), the engine immediately knows the MV is semantically stale or requires validation, even if the underlying data files haven't changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3967385004
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple path in the dependency graph.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its view metadata and one ore more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
Review Comment:
`snapshot_id` does not cover all alterations on the source table.
Renaming a column is a metadata only operation and does not incur a snapshot.
However, such operations can actually make a materialized view potentially stale (`SELECT * FROM source_table` still works, but delivers old column names).
Has there been given any thought about using the `metadata-file-location` instead of the `snapshot-id` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3966266072
##########
format/view-spec.md:
##########
@@ -322,3 +449,198 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+- `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+- `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+- `C` (regular view or materialized view, varies by strategy): `SELECT ... FROM F JOIN G ON ...`
+- `D` (regular view or materialized view, varies by strategy): `SELECT ... FROM H WHERE ...`
+- `E`, `F`, `G`, `H`: source Iceberg tables
+
+### Strategy 1: Empty refresh state (recency only)
Review Comment:
Well, having a flag for completeness means having a definition of completeness. And because it is very difficult to find a definition that suits every query engine we went with the formulation: "The producer selectively chooses a subset of the dependencies".
I totally agree that such a flag would be useful but due to the interoperability it is not in scope for the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4144104269
##########
format/view-spec.md:
##########
@@ -63,11 +75,13 @@ The view version metadata file has the following fields:
| _required_ | `versions` | A list of known [versions](#versions) of the view [1] |
| _required_ | `version-log` | A list of [version log](#version-log) entries with the timestamp and `version-id` for every change to `current-version-id` |
| _optional_ | `properties` | A string to string map of view properties [2] |
+| _optional_ | `max-staleness-ms` | The maximum time interval in milliseconds during which changed source table snapshots are considered fresh enough to skip refreshing [3] |
Review Comment:
We removed "max-staleness" from the current spec as we couldn't agree on a common definition across all engines. That said, every engine can still apply it's own staleness window it's just not standardized across engines and it has to define it's own properties for it.
Because it's such a common case we still list it as an example in the current version. We encourage it but we don't introduce a common definition that has to be enforced by every engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4101718067
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+* **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+* **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple paths in the dependency graph.
+* **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+* **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+* **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
Review Comment:
>Is the intent that a missing/expired snapshot reference should always be treated as stale
It doesn't have to always be stale. The consumer can implement its own policies. If its policy requires zero staleness then a missing snapshot reference probably means stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4285367429
##########
format/view-spec.md:
##########
@@ -322,3 +430,206 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "catalog" : "prod",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+A writer's choice of what to record varies along two independent dimensions:
+
+* Query evaluation — how the writer evaluates the materialized view’s query, in particular whether the result of nested materialized views is determined by expanding their own dependencies or consuming directly from the storage table. This determines the set of dependencies that could be recorded.
+* Coverage — which of those dependencies are actually recorded; a writer may record all, a subset, or none.
+
+The strategies below illustrate common combinations against the same shared query. Strategies 2 and 3 differ along structure; Strategies 1, 4, and 5 illustrate coverage choices.
+
+### Shared query
+
+* `A` (the materialized view being refreshed): `SELECT B.customer_id, C.category, SUM(B.amount) AS total FROM B JOIN C ON B.product_id = C.product_id GROUP BY B.customer_id, C.category`
+* `B` (regular view): `SELECT E.customer_id, E.product_id, E.amount FROM E JOIN D ON E.customer_id = D.customer_id`
+* `C` (materialized view): `SELECT F.product_id, G.category FROM F JOIN G ON F.category_id = G.category_id`
+* `D` (materialized view): `SELECT customer_id FROM H WHERE status = 'active'`
+* `E` (orders), `F` (products), `G` (categories), `H` (customers): source Iceberg tables
+
+### Strategy 1: Empty refresh state (recency only)
+
+The producer leaves `source-states` empty and relies entirely on `refresh-start-timestamp-ms`. Readers reuse the storage table based on a recency policy alone.
+
+`A`'s refresh state:
+
+```json
+{
+ "view-version-id": 1,
+ "refresh-start-timestamp-ms": 1573518435000,
+ "source-states": []
+}
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized view storage tables. The producer stops at each MV boundary and records the MV's storage table snapshot ID and view version ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14, version-id: 9
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12, version-id: 7
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query recursively expands the view definitions of the materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is **not** recorded.
+
+`C` and `D` are materialized views, treated as views (expanded).
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV — expanded as view] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [MV — expanded as view] <-- recorded in A: version-id: 7
+ ├── F [TABLE] <-- recorded in A: snapshot-id: 102
+ └── G [TABLE] <-- recorded in A: snapshot-id: 103
+```
+
+The recorded shape matches Strategy 1. The difference is semantic: `C` and `D` are materialized views whose view definitions were expanded; their storage tables are not part of the recorded state.
+
+### Strategy 4: Track only view versions
+
+The producer treats the storage table as reusable as long as the view definitions in the dependency chain are unchanged. Underlying table changes do not affect freshness. Only view version IDs are recorded.
+
+`C` and `D` are regular views.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] (not recorded)
+│ └── D [VIEW] <-- recorded in A: version-id: 9
+│ └── H [TABLE] (not recorded)
+└── C [VIEW] <-- recorded in A: version-id: 7
+ ├── F [TABLE] (not recorded)
+ └── G [TABLE] (not recorded)
+```
+
+Snapshots of `E`, `F`, `G`, `H` are not recorded. Reuse is sensitive to view-definition changes but insensitive to data changes in the underlying tables.
+
+### Strategy 5: Skip non-Iceberg dependencies
+
+The producer records only Iceberg sources and omits non-Iceberg dependencies entirely. Useful when the view query reads from a mix of Iceberg and non-Iceberg sources and the producer chooses to track only the Iceberg side.
Review Comment:
producer -> writer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "hashhar (via GitHub)" <gi...@apache.org>.
hashhar commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2048363317
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
Review Comment:
What are some reasons why the catalog and namespace should not be inherited from the MV's catalog and namespace?
IIUC the proposal allows:
- catalog1.ns1.source_table (source table)
- catalog2.ns2.mv (materialized view)
- catalog3.ns3.storage_table (storage table)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2457420431
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables, views, and materialized views in the materialized view's fully expanded query tree at refresh time. Source table states are stored in `source-table-states` and source view states in `source-view-states`. For source views, `source-view-states` includes indirect references — tables or views nested within other views but not directly referenced in the query. For source materialized views, both the source view and its storage table are included in the refresh state. Indirect references are excluded for materialized view sources; query engines may recursively expand the query tree to determine freshness. The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#source-table) records for all tables that are directly or indirectly referenced in the materialized view query |
+| _required_ | `source-view-states` | A list of [source view](#source-view) records for all views that are directly or indirectly referenced in the materialized view query |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source table
+
+A source table record captures the state of a source table at the time of the last refresh operation.
Review Comment:
the state of a source table (including source MV's storage table)?
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables, views, and materialized views in the materialized view's fully expanded query tree at refresh time. Source table states are stored in `source-table-states` and source view states in `source-view-states`. For source views, `source-view-states` includes indirect references — tables or views nested within other views but not directly referenced in the query. For source materialized views, both the source view and its storage table are included in the refresh state. Indirect references are excluded for materialized view sources; query engines may recursively expand the query tree to determine freshness. The refresh state has the following fields:
Review Comment:
> For source views, `source-view-states` includes indirect references — tables or views nested within other views but not directly referenced in the query.
Should we clarify as
```
tables or views nested within other views but not directly referenced in the query.
-->
views nested within other views (excluding MVs) that are not directly referenced in the query
```
-------------------------------
nit: move `For source materialized view, ...` to a separate paragraph to highlight the special clarification for source MV?
-------------------------------
query engines may recursively expand the query tree to determine freshness
->
during read time, query engines may recursively expand the query tree to determine freshness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2783210233
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+Consumers should only read from the storage table if the materialized view is "fresh" and therefore adequately represents the logical query definition of the view.
+Different systems define freshness differently based on time-based and logical factors.
+
+**Time-based freshness (consumer-defined):**
+
+Consumers may apply time-based freshness policies, such as allowing a certain staleness window based on `refresh-start-timestamp-ms`.
+When evaluating freshness, consumers:
+- Must first evaluate their own time-based freshness policy.
+- May additionally compare the `source-states` list against the states loaded from the catalog to verify the producers logical freshness policy.
+- May parse the view definition to implement a more sophisticated policy.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Must not read from the storage table when the materialized view doesn't meet freshness criteria.
+
+**Logical freshness (producer-defined):**
Review Comment:
Defining Logical freshness and time-based freshness is confusing IMO .
I think the plan to acknowledge wide range of policies WRT to freshness ( share some examples)
and define Refresh state as "standardize means" of communicating metadata between consumer and producer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2628240131
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
+
+Different systems interpret freshness differently, typically based on the objects referenced in the fully expanded query tree of the materialized view. Some systems consider only direct children, others only leaf nodes, and some the entire query tree. The specific interpretation is determined by the producer of the storage table.
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time. It contains a list of directly or indirectly referenced source states that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+
+**Consumer evaluation:**
+- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`.
+- The consumer may additionally compare the `source-states` list against the states loaded from the catalog.
+- The consumer trusts that the producer has provided all states necessary to determine freshness.
Review Comment:
I am not sure this bullet point about `trust` is needed.
During the meeting, we also discussed more flexibility for consumers sth along this line.
```
- The consumer may parse the view definition to implement more sophisticated policy.
```
##########
format/view-spec.md:
##########
@@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+
## Specification
### Terms
* **Schema** -- Names and types of fields in a view.
* **Version** -- The state of a view at some point in time.
+* **Storage table** -- Iceberg table that stores the precomputed data of a materialized view.
+* **Source table** -- A table reference that occurs in the query definition of a materialized view. The materialized view depends on the data from the source tables.
+* **Source view** -- A view reference that occurs in the query definition of a materialized view. The materialized view depends on the definitions from the source views.
Review Comment:
for consistency, maybe
```
Source materialized view -- A materialized view reference that occurs in the query definition of a materialized view.
```
I actually think we can remove the part from the bullet points, as I don't see it clarify anything additionally
```
The materialized view depends on the data from the source tables.
```
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
+
+Different systems interpret freshness differently, typically based on the objects referenced in the fully expanded query tree of the materialized view. Some systems consider only direct children, others only leaf nodes, and some the entire query tree. The specific interpretation is determined by the producer of the storage table.
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time. It contains a list of directly or indirectly referenced source states that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+
+**Consumer evaluation:**
+- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`.
Review Comment:
we probably need to clearly the evaluation, like
```
A materialized view is fresh if `refresh-start-timestamp-ms` is within the window `[now - max-staleness-ms, now]`
```
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
Review Comment:
> A materialized view is considered fresh when its precomputed data is usable by consumers.
This definition doesn't capture what should be considered fresh. Later part on consumer responsibilities is kind of defining what fresh mean.
##########
format/view-spec.md:
##########
@@ -63,11 +75,13 @@ The view version metadata file has the following fields:
| _required_ | `versions` | A list of known [versions](#versions) of the view [1] |
| _required_ | `version-log` | A list of [version log](#version-log) entries with the timestamp and `version-id` for every change to `current-version-id` |
| _optional_ | `properties` | A string to string map of view properties [2] |
+| _optional_ | `max-staleness-ms` | The maximum time interval in milliseconds during which changed source table snapshots are considered fresh enough to skip refreshing [3] |
Review Comment:
the wording here isn't super clear how this config should be used. It also doesn't capture the delayed view semantic that @igorbelianski-cyber mentioned.
We should remove `to skip refreshing` part, as consumers can have other fallback behaviors like fail or treat MV as a logical view
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
Review Comment:
We should add a paragraph to describe possible consumer behavior is a MV is considered stale.
```
- Can fail, or refresh inline, or treat MV as a logical view
- Mustn’t consume from the storage table
```
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
+
+Different systems interpret freshness differently, typically based on the objects referenced in the fully expanded query tree of the materialized view. Some systems consider only direct children, others only leaf nodes, and some the entire query tree. The specific interpretation is determined by the producer of the storage table.
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time. It contains a list of directly or indirectly referenced source states that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+
+**Consumer evaluation:**
+- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`.
+- The consumer may additionally compare the `source-states` list against the states loaded from the catalog.
+- The consumer trusts that the producer has provided all states necessary to determine freshness.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Materialized views can reference source objects of different types, such as Iceberg tables and views. Source state records have a common field `type` that determines the form, which can be one of the following:
+
+* `table`: An Iceberg table
+* `view`: An Iceberg view
Review Comment:
I am wondering if source MV should be one entry with combined view and storage table status in this single list. It is related to my question on how to load MV in REST catalog. Can we hide the storage table to consumers for access control. Only define access control at view level and storage table hiding would auto inherit the view access policy? but it could be a problem for non REST catalog.
If a single entry, source MV can have the following states
- view-uuid
- view-version-id
- storage-table-uuid
- storage-table-snapshot-id
`ref` field for source table state is not applicable to storage table.
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
+
+Different systems interpret freshness differently, typically based on the objects referenced in the fully expanded query tree of the materialized view. Some systems consider only direct children, others only leaf nodes, and some the entire query tree. The specific interpretation is determined by the producer of the storage table.
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time. It contains a list of directly or indirectly referenced source states that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+
+**Consumer evaluation:**
+- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`.
+- The consumer may additionally compare the `source-states` list against the states loaded from the catalog.
Review Comment:
We probably lack a bit of details here. If the coarse-grained evaluation considered the MV stale and this evaluation considered it fresh, the MV should be considered fresh. But such priority is not expressed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-3658242606
Benny,
Are you proposing to simplify states from
tri-state (fresh, stale ,invalid ). down to 2 states (fresh, stale) ?
with various suggested above flavors implementing freshness
determination ?
Personally I would prefer the simplification (I am not certain i
interpreted your comment correctly)
PS
If we were to pursue this option we would need to make some definitive
statements about not initialized MVs.
On Fri, Dec 12, 2025 at 8:46 PM Benny Chow ***@***.***> wrote:
> ***@***.**** commented on this pull request.
> ------------------------------
>
> In format/view-spec.md
> <https://github.com/apache/iceberg/pull/11041#discussion_r2616065681>:
>
> > +When `ref` is `null` or not set, it defaults to "main".
> +
> +#### Source view
> +
> +A source view record captures the state of a source view at the time of the last refresh operation.
> +
> +| Requirement | Field name | Description |
> +|-------------|----------------|-------------|
> +| _required_ | `uuid` | The uuid of the source view |
> +| _required_ | `version-id` | Version-id of when the last refresh operation was performed |
> +
> +#### Status Interpretation
> +
> +During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
> +
> +* **invalid** -- The current `version_id` of the materialized view does not match the `view-version-id` recorded in its refresh state. A read operation cannot proceed using the materialized view's data.
>
> I personally find trying to use invalid, fresh and stale statuses as too
> generic. We could instead describe the 3 high level ways a consumer would
> validate the freshness:
>
> 1. Use refresh-start-timestamp-ms
> 2. Use source-table-states and source source-view-states. Optionally
> validating nested MV refresh-state..
> 3. Advanced validation through view SQL expansion and refresh-state
> comparison.
>
> For 3, the consumer may determine that affected partitions in the table
> are still up to date after pushing down filters.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/iceberg/pull/11041#discussion_r2616065681>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B3NEUQTQP2XE7PBFVSAGYE34BOKY7AVCNFSM6AAAAABNKKAL3KVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTKNZUGE3TOMBQGA>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "sfc-gh-ibelianski (via GitHub)" <gi...@apache.org>.
sfc-gh-ibelianski commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2593951478
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of all the source tables, OR all source table snapshots that differ from the last refresh have timestamps within a configured staleness window.
Review Comment:
I think during the last meeting we have agreed to use "delayed view definition/semantics" as it doesn't mandate optimizations and allows for false negatives (allows claiming that view is stale as freshness logic is naive ). The wording above mandates very specific optimization of additional checks needed if refresh is outside of the staleness window.
proposal:
Invalid: The current version_id of the materialized view does not match the view-version-id recorded in its refresh state. A read operation cannot proceed using the materialized view's data.
Fresh: if it is Valid and the stored data represents the result set that would have been retrieved if the underlying View Query was executed at some point during the defined Staleness Window.
stale: if it is valid but not Fresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2590268273
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of all the source tables, OR all source table snapshots that differ from the last refresh have timestamps within a configured staleness window.
+* **stale** -- The `snapshot_id`s do not match for at least one source table and at least one differing snapshot has a timestamp outside the configured staleness window, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
> at least one source table and at least one differing snapshot
let's emphasize the word **and**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2612328840
##########
format/view-spec.md:
##########
@@ -160,6 +177,71 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time.
+
+* Source view states are stored in `source-view-states`. It includes indirect references — views nested within other views (excluding MVs).
Review Comment:
i think in the meeting we were converging on including MVs here and below. Lets revisit during next meeting to make sure there is no ambiguity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2629898567
##########
format/view-spec.md:
##########
@@ -63,11 +75,13 @@ The view version metadata file has the following fields:
| _required_ | `versions` | A list of known [versions](#versions) of the view [1] |
| _required_ | `version-log` | A list of [version log](#version-log) entries with the timestamp and `version-id` for every change to `current-version-id` |
| _optional_ | `properties` | A string to string map of view properties [2] |
+| _optional_ | `max-staleness-ms` | The maximum time interval in milliseconds during which changed source table snapshots are considered fresh enough to skip refreshing [3] |
Review Comment:
I've added the delayed view semantics to the Notes section below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2631557404
##########
format/view-spec.md:
##########
@@ -160,6 +178,108 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view's precomputed data becomes stale as the tables and views referenced in its query definition change over time. Freshness determines whether the precomputed data accurately represents the logical query definition at the current state of its dependencies.
+
+Different systems define freshness differently, based on how much of the dependency graph must be current. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, "fresh" can mean strict end-to-end consistency, acceptable lag, or policy/version compliance.
+
+A materialized view is considered fresh when its precomputed data meets the freshness criteria defined by the consumer's evaluation policy. When these criteria are not met, the materialized view is considered stale.
+
+#### Refresh state
+
+The refresh state record captures the unique dependencies in the materialized view's dependency graph. These dependencies include source Iceberg tables, views, and nested materialized views that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
Review Comment:
I think the intent to always treat the data as fresh should be specified with `max-staleness-ms` set to `null`. If that is what you mean with doesn't require staleness determination.
The issue is that with an emtpy list the behavior for "cannot be determined" and "doesn't need to be determined" is different. In case of "cannot be determined" the consumer should use the coarse-grained check. But with hte "doesn't need to be determined" the consumer should just treat the data as fresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2631490048
##########
format/view-spec.md:
##########
@@ -160,6 +178,108 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view's precomputed data becomes stale as the tables and views referenced in its query definition change over time. Freshness determines whether the precomputed data accurately represents the logical query definition at the current state of its dependencies.
+
+Different systems define freshness differently, based on how much of the dependency graph must be current. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, "fresh" can mean strict end-to-end consistency, acceptable lag, or policy/version compliance.
+
+A materialized view is considered fresh when its precomputed data meets the freshness criteria defined by the consumer's evaluation policy. When these criteria are not met, the materialized view is considered stale.
+
+#### Refresh state
+
+The refresh state record captures the unique dependencies in the materialized view's dependency graph. These dependencies include source Iceberg tables, views, and nested materialized views that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+
+**Consumer evaluation:**
+- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`. A materialized view is fresh if `refresh-start-timestamp-ms` is within the window `[now - max-staleness-ms, now]`.
+- The consumer may additionally compare the `source-states` list against the states loaded from the catalog. If this evaluation determines the materialized view is fresh, it overrides the coarse-grained evaluation result.
+- The consumer may parse the view definition to implement a more sophisticated policy.
+- When a materialized view is considered stale, the consumer can fail, refresh inline, or treat the materialized view as a logical view. The consumer must not consume from the storage table when the materialized view is stale.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Materialized views can reference source objects of different types, such as Iceberg tables, view, and materialized views. Source state records have a common field `type` that determines the form, which can be one of the following:
+
+* `table`: An Iceberg table
+* `view`: An Iceberg view
+* `materialized-view`: An Iceberg materialized view
+
+The metadata fields for each type are defined below:
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
Review Comment:
as i mentioned above we may nee path here or formulate some policy(producer mus store the oldes snapshot id for the duplicate objects )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1852962681
##########
format/view-spec.md:
##########
@@ -42,12 +42,27 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view that precompute the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and information about the refresh state to determine if the data is still fresh.
+The refresh state is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
Review Comment:
Nit: how about new paragraph for 'refresh state'
and then bullet point for each state
Also , is refresh state the same as storage table state? Its a bit confusing to read as is, can use the same terminology if its the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2497435997
Yes correct, the idea is to filter-out the storage-table for the `listTables` operation. But I would regard this as not part of the table/view spec but should rather be included in the REST catalog specification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2581043683
> > > If i understand correctly @wmoustafa comment on the mailing list, then there is some ambiguity here for what to put, if the same table in expressed in the various forms (catalog.database.name) or (database.name) or (name), either in same sql statement or in the different sql representations of the same view.
> >
> >
> > I am still -1 on this change before resolving #11365. Further, I am also -1 on referencing table identifiers in table snapshot summary. We had extended discussions on this and converged at some point already.
>
> This is from the discussion in https://lists.apache.org/thread/v8m1tpb91g740gmvqyphhjw37mpr8sl7 right? if i understand the discussion, it is good to clarify the existing assumptions that allow portability of all views (not just MV), I'm also +1 on adding the idea of #11365, but it seems most opinions there was that it should not block MV spec.
I think it should block, but in all cases, we should have a mailing list vote on this spec anyways. That way it becomes formal. Happy to continue the discussion before voting to align.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1909298744
##########
format/view-spec.md:
##########
@@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [full identifier](#full-identifier) of the storage table |
Review Comment:
The table spec doesn't have a term as `full identifier`. we can probably keep it generic as `table identifier`.
This also matches the fact that catalog name may be skipped in the table identifier and use view catalog or default catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1744845733
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
Review Comment:
That would work for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2530559251
> If i understand correctly @wmoustafa comment on the mailing list, then there is some ambiguity here for what to put, if the same table in expressed in the various forms (catalog.database.name) or (database.name) or (name), either in same sql statement or in the different sql representations of the same view.
I am still -1 on this change before resolving https://github.com/apache/iceberg/pull/11365. Further, I am also -1 on referencing table identifiers in table snapshot summary. We had extended discussions on this and converged at some point already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1956953050
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the views' catalog |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+A property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. Indirect references are the tables/views that are not directly referenced in the query but are nested within other views. The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#source-table) records for all tables that are directly or indirectly referenced in the materialized view query |
+| _required_ | `source-view-states` | A list of [source view](#source-view) records for all views that are directly or indirectly referenced in the materialized view query |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source table
+
+A source table record captures the state of a source table at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
Review Comment:
I think any ambiguity we have with table identifiers in the refresh state we will have with the view's storage table. Its probably best to leave the table identifiers out of the refresh state and not contribute to more view interoperability problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3040992649
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized views' |
Review Comment:
The idea to leave the catalog field optional was to sidestep the catalog alias problem for use cases that involved only one catalog. For that use case, different query engines could use different catalog aliases and still be able read and refresh materialized views. I agree that this is not a clean solution.
Instead we should use fully qualified identifiers and require consistent environments for query engine interoperability. Meaning the query engine environments have to be setup such that the same catalog identifiers can be used across different query engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3034990232
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized views' |
Review Comment:
@wmoustafa I agree we have a resolution problem with tables from intermediary views. I would prefer to solve this problem by requiring the source table states to be fully qualified - i.e. the catalog is no longer optional. This leave no ambiguity in terms of what table the producer used.
This means that producer and consumer engines will either have to agree on the same catalog name or the consumer is able to re-map the catalog name themselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3011290321
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
+| _optional_ | `ref` | Branch name of the source table being referenced in the view query |
+
+When `ref` is `null` or not set, it defaults to "main".
+
+#### Source view state
+
+A source view record captures the state of a source view at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `view` |
+| _required_ | `name` | A string specifying the name of the source view |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
Review Comment:
```suggestion
| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized views' |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3010718573
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
> The storage table must be in the same catalog as the materialized view.
Do we actually need to make this a requirement? I understand that it can make things complicated, but I don't see a technical reason we need to assert this requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3011245475
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
Review Comment:
Great point, makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3011280826
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
Review Comment:
```suggestion
| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized views' |
```
Prefer not to callout `null` specifically because both null and omitted are valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3319168809
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
Review Comment:
```suggestion
A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh and should not be consumed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3320521206
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query.
Review Comment:
I'm OK with that too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3319257256
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies in the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an upstream materialized view's storage table) and/or a `view` entry. Upstream materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track (diamond dependency pattern).
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the freshness of the storage table:
Review Comment:
```suggestion
Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
```
This line introduces the issue of multiple definitions of "freshness". We need to relax this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3320624687
##########
format/view-spec.md:
##########
@@ -322,3 +449,215 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+- `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+- `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+- `C` (regular view or materialized view, varies by strategy): `SELECT ... FROM F JOIN G ON ...`
+- `D` (regular view or materialized view, varies by strategy): `SELECT ... FROM H WHERE ...`
+- `E`, `F`, `G`, `H`: base Iceberg tables
+
+### Strategy 1: Track all nested tables and views (no nested MVs)
+
+The view query reads only base tables and regular views. The refresh state tracks snapshot IDs of all deeply nested base tables and version IDs of all views traversed. Reuse of the storage table is sensitive to changes in any of them.
+
+`C` and `D` are regular views.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [VIEW] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [VIEW] <-- recorded in A: version-id: 7
+ ├── F [TABLE] <-- recorded in A: snapshot-id: 102
+ └── G [TABLE] <-- recorded in A: snapshot-id: 103
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
Review Comment:
In this example, I think it's more logical to have included C and D in the refresh state as well. Basically, if a MV directly references a view regardless if the view is used as a logical view or nested MV, I would have expected the view to be in the refresh state. I wonder what others think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2381352562
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of all the source tables.
+* **stale** -- The `snapshot_id`s do not match for at-least one source table, indicating that a refresh operation needs to be performed to capture the latest source table changes.
+* **invalid** -- The current `version_id` of the materialized view does not match the `refresh-version-id` of the refresh state.
Review Comment:
Thanks for the catch, it's supposed to be `view-version-id`. This was changed during the discussion. I'll update it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2288758854
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of the source tables.
+* **stale** -- The `snapshot_id`s do not match, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
I put together a doc on the two options regarding source MVs with pros and cons. please take a look.
https://docs.google.com/document/d/1_StBW5hCQhumhIvgbdsHjyW0ED3dWMkjtNzyPp9Sfr8/edit?tab=t.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1741391714
##########
format/view-spec.md:
##########
@@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
+The storage table can have the states "fresh", "stale" or "invalid".
Review Comment:
I don't see a property for this state on the storage table? I also don't think we need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2985974059
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
Review Comment:
What is the added value of having the storage table exposed in the metastore?
For reference, in Trino Materialized Views the storage tables are not visible in the metastore.
https://github.com/trinodb/trino/pull/18853
The way that the content of the MV is being stored is more of an "implementation detail" and not something end users should know about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3003387100
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
Review Comment:
This field separates logical view (current) with materialized view. MV has this extra field.
When the storage table should be exposed in the REST catalog is out of the scope of the MV spec. catalog can choose to skip storage tables for list tables call for consumers. MV producer would still need to be able to load and update the storage table in the catalog.
There is this problem of number of round trips for loading view or MV in REST spec
* loading a logical view requires 2 round trips: (1) load table - failed (2) load view
* load a MV requires 3 round trips with current REST spec: (1) load table -failed (2) load view (3) load storage table
I am working on a proposal of universal load endpoint for relational objects: table, view, materialized view. One round trip should be able to load all 3 types of relational objects. I probably can share next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3033423397
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
Thread discussing same type of problem with child table identifiers: https://github.com/apache/iceberg/pull/11041#discussion_r3033212803.
This thread would be simplified as well.
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
@stevenzwu We did have cross-catalog child tables at LinkedIn, so this is a pattern I've seen in practice. That said, I agree it's cleaner to avoid it. Constraining child tables to the same catalog as the view would significantly simplify the spec, particularly around identifier resolution and freshness checking. I'd be supportive of that constraint. The limitation is small and not worth the complexity it introduces.
**Worth noting:** this constraint doesn't prevent querying across data sources. A single catalog can still federate access to BigQuery, Snowflake, or other systems. The catalog is just an alias; same-catalog doesn't mean same-storage. @talatuyarer FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3239074944
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
Review Comment:
If the `refresh-state` is to be filled arbitrarily by the producers, what is the rationale of mentioning it in the spec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242655304
##########
format/view-spec.md:
##########
@@ -190,92 +190,93 @@ The table identifier for the storage table that stores the precomputed results.
### Storage table metadata
This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
-The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
| Requirement | Field name | Description |
|-------------|-----------------|-------------|
| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
#### Freshness
-A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
-Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
-**Consumer behavior:**
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
-When evaluating freshness, consumers:
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
-- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
-- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
-- May parse the view definition to implement more sophisticated policies.
-- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
-- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+##### Producer flexibility
-**Producer behavior:**
-
-Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
-Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
-Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
When writing the refresh state, producers:
-- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
-- If the source state cannot be determined for all objects (for example, for non-Iceberg tables or non-deterministic functions) may leave the source states list empty.
-- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
Review Comment:
Agree Should make more sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3244343363
##########
format/view-spec.md:
##########
@@ -190,92 +190,93 @@ The table identifier for the storage table that stores the precomputed results.
### Storage table metadata
This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
-The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
| Requirement | Field name | Description |
|-------------|-----------------|-------------|
| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
#### Freshness
-A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
-Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
Review Comment:
i like upstream better ( it still preserves ambiguity , but we can use direct and indirect upstream for the cases when ambiguity needs be resolved)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242586216
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
Review Comment:
We're over specifying here. This statement is now defining how state must be calculated, not just what we consider 'fresh' to mean. I would get rid of the second sentence because we do not need to define the rest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4094119181
##########
format/view-spec.md:
##########
@@ -322,3 +449,198 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
Review Comment:
Agreed the MV is genuinely stale once the comment is added — I'd keep the freshness rule strict on view-version-id. But this is where a "smart" REFRESH MATERIALIZED VIEW helps: the refresh can detect that only a column comment changed and that the rest of the version and the precomputed data are still valid, then write a new refresh-state with the updated view-version-id without recomputing the data. The MV is fresh again after that. The tradeoff is that users may need to run a REFRESH in such cases to update the metadata — which avoids having to define/parse semantic SQL equivalence in the spec (your reconciliation layer's job) on the consumer side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4064093589
##########
format/view-spec.md:
##########
@@ -322,3 +449,198 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
Review Comment:
i'm building a reconciliation layer to verify such whether the view sql and schema are still the same despite the fact that version id has evolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4144228023
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
Review Comment:
I think we should discuss the term "serving policy" in the sync with more people involved. The tension between a precise and vague definition has existed for a while. And we need to make sure everybody is on board.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4283412678
##########
format/view-spec.md:
##########
@@ -322,3 +430,206 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "catalog" : "prod",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+A writer's choice of what to record varies along two independent dimensions:
+
+* Query evaluation — how the writer evaluates the materialized view’s query, in particular whether the result of nested materialized views is determined by expanding their own dependencies or consuming directly from the storage table. This determines the set of dependencies that could be recorded.
+* Coverage — which of those dependencies are actually recorded; a writer may record all, a subset, or none.
+
+The strategies below illustrate common combinations against the same shared query. Strategies 2 and 3 differ along structure; Strategies 1, 4, and 5 illustrate coverage choices.
+
+### Shared query
+
+* `A` (the materialized view being refreshed): `SELECT B.customer_id, C.category, SUM(B.amount) AS total FROM B JOIN C ON B.product_id = C.product_id GROUP BY B.customer_id, C.category`
+* `B` (regular view): `SELECT E.customer_id, E.product_id, E.amount FROM E JOIN D ON E.customer_id = D.customer_id`
+* `C` (materialized view): `SELECT F.product_id, G.category FROM F JOIN G ON F.category_id = G.category_id`
+* `D` (materialized view): `SELECT customer_id FROM H WHERE status = 'active'`
+* `E` (orders), `F` (products), `G` (categories), `H` (customers): source Iceberg tables
+
+### Strategy 1: Empty refresh state (recency only)
+
+The producer leaves `source-states` empty and relies entirely on `refresh-start-timestamp-ms`. Readers reuse the storage table based on a recency policy alone.
+
+`A`'s refresh state:
+
+```json
+{
+ "view-version-id": 1,
+ "refresh-start-timestamp-ms": 1573518435000,
+ "source-states": []
+}
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID and view version ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14, version-id: 9
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12, version-id: 7
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query reads from materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is **not** recorded.
Review Comment:
> but the query reads from materialized views
but the query expands recursively the view queries corresponding to the materialized views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1741476428
##########
format/view-spec.md:
##########
@@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
+The storage table can have the states "fresh", "stale" or "invalid".
Review Comment:
What I mean is that if you check for the freshness of the table, the result can be "fresh", "stale", or "invalid". Maybe state is not the right term for it. And you can't assign a property to it. Maybe we can drop the sentence from the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1851820348
##########
format/view-spec.md:
##########
@@ -42,12 +42,25 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
Review Comment:
I think it's import to provide a little context about how Materialized Views are realized. I think it's much harder to understand the additional properties if the overall structure isn't clear. For example, I think it's very important to state that it is a combination of a view and a storage table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2085552249
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
Review Comment:
I support linking from Table => MV for this. I don't know if the information needs to be in both places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "hashhar (via GitHub)" <gi...@apache.org>.
hashhar commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2080999014
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
Review Comment:
Having a dedicated namespace makes sense. A separate catalog is what I can't think of a use for personally. If you can find the comments it would be helpful.
(I'm not asking for a change, as a person who has to possibly implement this downstream in engines I was thinking about use-cases).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2085437752
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
Review Comment:
```suggestion
Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1911104542
##########
format/view-spec.md:
##########
@@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [full identifier](#full-identifier) of the storage table |
Review Comment:
Just to double check. In a more general setting, the `Identifier` concept could also reference a view. If we go with `Table Identifier` this could either require an additional definition in the future or might be confusing when used for views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1870027645
##########
format/view-spec.md:
##########
@@ -82,9 +98,13 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [partial identifier](#partial-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When 'storage-table' is `null` or not set, the entity is a common view, otherwise it is a materialized view.
+The catalog of the storage table must be same as the one of the materialized view.
Review Comment:
@stevenzwu @JanKaul @danielcweeks Does this overlap with https://github.com/apache/iceberg/pull/11365? Can we discuss both in a wholistic way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1861263489
##########
format/view-spec.md:
##########
@@ -158,6 +174,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Partial identifier
+
+The partial identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the source table |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
+
+For that the additional field "refresh-state" is introduced as an opaque record in the table snapshot summary.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. It has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#soure-table) records for all tables that are directly or indirectly referenced in the materialized view query |
Review Comment:
maybe it is more clear to change `indirectly` to `transitively`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1960719638
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
Review Comment:
resurface a comment which might have been missed because it was combined with another minor comment. this is actually probably a little bigger decision.
should this whole section be added to the `Table Spec` page? we can add cross references btw the `Table Spec` and `View Spec` pages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2809837721
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and nested materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of nested materialized views |
+| `view` | An Iceberg view, including nested materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
Review Comment:
The binding is not strictly required... in the spirt of not trying to optimize the v1 spec, we don't have to include it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2237241408
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table and refresh metadata.
+Refresh metadata contains information about the "source tables", tables referenced in the query definition of the materialized view.
+The storage table can be "fresh", "stale" or "invalid", depending on the following the following situations:
Review Comment:
```suggestion
During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following the following situations:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2237234850
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table and refresh metadata.
Review Comment:
`and` would be misinterpreted that the refresh metadata is part of the view entity. `with` is more clear that the refresh state is part of the storage table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2289941916
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of the source tables.
+* **stale** -- The `snapshot_id`s do not match, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
Which option do you prefer? And should I include this in the spec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2791436538
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and nested materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of nested materialized views |
+| `view` | An Iceberg view, including nested materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
Review Comment:
I didn't quite get this comment. @igorbelianski-cyber can you elaborate a little?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2794941285
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and nested materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of nested materialized views |
+| `view` | An Iceberg view, including nested materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
Review Comment:
I agree with our scenario. If it used the storage table, the storage table will be in the refresh-state. If it used the view definition it will need to include some other downstream tables in the refresh-state. Depending on the interpretation of the producer.
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and nested materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of nested materialized views |
+| `view` | An Iceberg view, including nested materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
Review Comment:
I agree with your scenario. If it used the storage table, the storage table will be in the refresh-state. If it used the view definition it will need to include some other downstream tables in the refresh-state. Depending on the interpretation of the producer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2795132238
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and nested materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of nested materialized views |
+| `view` | An Iceberg view, including nested materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
Review Comment:
clarification for the MV that depends on V1, MV1, T1
would have table dependencies : T1, MV1.table
view dependencies MV1 , V1
there is no way to tell that MV1 and MV1.table are related (without querying for more data)
I was askin whether that binding ned to be exposed ( I do not have strong scenario to require that ) just sanity check , getting feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2631410916
##########
format/view-spec.md:
##########
@@ -42,12 +42,25 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
Review Comment:
this reads like "Source materialized view" excluded from the list on purpose ( i remember use agreeing that source materialized view should be included with precise spelling out of what it means)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2631584367
##########
format/view-spec.md:
##########
@@ -160,6 +178,108 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view's precomputed data becomes stale as the tables and views referenced in its query definition change over time. Freshness determines whether the precomputed data accurately represents the logical query definition at the current state of its dependencies.
+
+Different systems define freshness differently, based on how much of the dependency graph must be current. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, "fresh" can mean strict end-to-end consistency, acceptable lag, or policy/version compliance.
+
+A materialized view is considered fresh when its precomputed data meets the freshness criteria defined by the consumer's evaluation policy. When these criteria are not met, the materialized view is considered stale.
+
+#### Refresh state
+
+The refresh state record captures the unique dependencies in the materialized view's dependency graph. These dependencies include source Iceberg tables, views, and nested materialized views that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
Review Comment:
the scenario is "freshness needs to be determined" and "coarse grain check deemed to be sufficient" (list is superfluous in this case)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3030402201
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
Review Comment:
for freshness evaluation, I don't see the benefit or value of storing two versions of the same table. Doesn't really matter if the snapshot has expired or not. If the table snapshot in the refresh-state and the latest data-changing snapshot from catalog are different, then consumers can consider the storage table stale. For that purpose, the oldest snapshot would suffice.
On the other hand, I also don't see a strong reason to block such behavior. Original I thought the producer needs to update just one entry and hence would need lineage information to distinguish the entries. During today's meeting, this will not be an issue as long as producers update both entries together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "InvisibleProgrammer (via GitHub)" <gi...@apache.org>.
InvisibleProgrammer commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-3709796824
Hi, I'm not an Iceberg contributor, I'm just heavily interested in implementing this feature in Hive.
Please allow me to add a couple of questions about this specification (And also, please correct me in case I'm interpreting something wrong):
- There is a field in view metadata called `format-version`. The spec says it should be 1. Can the format version be changed with this specification?
- I was thinking about the storage table: at this point if we pick up a table, we just don't know if it is a storage table of a materialized view or just a regular table. Can the table metadata get a reference to the materialized view in case it is a storage table of a view? It would help other engines to protect the storage table from accidental table operations: if the engine is more strict, it can completely hides that table or disallow to do DML operations on that. If the engine less strict it can completely ignore that field.
And lastly please let me bring up a question about the timeline: as I see, this discussion goes for more than a year. I wonder, is there a way to speed the planning phase up? Iceberg is well known as a project evolving fast. I know there are still open questions and fine tuning. What about making an agreement on a minimal version that can work, saying that is the format-version 2 accept it, let the engines implement their part and making an agreement about the final, well-detailed version (maybe format-version 3) after gathering experience of the version 2?
I really appreciate your hard work, I believe materialized view will be a great Iceberg feature.
Thank you,
Zsolt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3465059980
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized views' |
Review Comment:
The spec says:
```
| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
```
So if the catalog entry is null, it defaults to the materialized view's catalog and not the `default-catalog` from the view. The engine that refreshes the MV will fully resolve all the identifiers, but in case the catalog-name is the same as the MV's catalog-name, it is not required to include it in the source-states.
That said, I'm okay with requiring fully resolved identifiers in the source-states. It was just a measure to solve the "catalog alias" problem at the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3019867183
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
+| _optional_ | `ref` | Branch name of the source table being referenced in the view query |
+
+When `ref` is `null` or not set, it defaults to "main".
+
+#### Source view state
+
+A source view record captures the state of a source view at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `view` |
+| _required_ | `name` | A string specifying the name of the source view |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
+| _required_ | `uuid` | The uuid of the source view |
+| _required_ | `version-id` | The version-id of the source view that was read during the refresh operation |
+
+#### Storage table creation and configuration
+
+When processing a `CREATE MATERIALIZED VIEW` statement, query engines must:
+
+1. Create the storage table as a regular Iceberg table with any specified configurations (partitioning, sort order, compression, etc.).
Review Comment:
I think this CREATE MATERIALIZED VIEW detail is helpful. I'm not reading it as something that the engine must do or must not do with the table configuration. Maybe change "query engine must" to "query engine should" to lessen the tone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3284699930
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+A snapshot whose refresh state violates a `Must` rule is invalid; consumers may treat it as if it had no `refresh-state`.
+
+##### Consumer options
+
+Consumers may use any combination of the following to assess the storage table:
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for intermediate materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table; otherwise it evaluates the view query in place of the storage table.
Review Comment:
or describe (but not limited to) some valid behaviors. the current wording can imply a single valid behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3014525620
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
Review Comment:
With regards to freshness, to argument was as follows:
- all source tables need to be up-to-date for a storage table to be fresh
- if any source table snapshot is out-of-date, the oldest snapshot is also out-of-date
- if the oldest snapshot is up-to-date, all other snapshots are up-to-date
- the oldest snapshot is enough information to determine freshness in the diamond pattern
Your comment is really relevant for incremental processing though. As the information what data has been processed is vital. I think we should then include all source table states, even different snapshots of the same table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3014787371
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
Review Comment:
@sfc-gh-ibelianski @stevenzwu @bennychow What do you think about including all distinct source states in case of the diamond pattern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3029079829
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
Review Comment:
i think we discussed as this being optional change =>. /have to be/. ===>. /should be/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4062391973
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple path in the dependency graph.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its view metadata and one ore more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
Review Comment:
Thanks for bringing this up, we don't cover this case yet. Personally, I would like to keep the "snapshot-id" field "required" to semantically convey that it should be populated whenever possible. Having it as "optional" opens room for discussion on when to actually populate it.
Since this is typically a temporary edge case (when tables have no data in them) I would opt for not recording empty tables in the refresh state.
CC @bennychow @stevenzwu @wmoustafa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4064083299
##########
format/view-spec.md:
##########
@@ -322,3 +449,198 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
Review Comment:
```
String sourceTableName = "test_comment_freshness_source_" + randomNameSuffix();
String materializedViewName = "test_comment_freshness_mv_" + randomNameSuffix();
assertUpdate("CREATE TABLE " + sourceTableName + " (a bigint)");
assertUpdate("INSERT INTO " + sourceTableName + " VALUES 1", 1);
assertUpdate("CREATE MATERIALIZED VIEW " + materializedViewName + " AS SELECT * FROM " + sourceTableName);
assertUpdate("REFRESH MATERIALIZED VIEW " + materializedViewName, 1);
assertFreshness(materializedViewName, "FRESH");
// COMMENT ON COLUMN for an Iceberg view creates a new version for the view
assertUpdate("COMMENT ON COLUMN " + materializedViewName + ".a IS 'some comment'");
// FIXME https://github.com/trinodb/trino/pull/31027#discussion_r4026696841
// getMaterializedViewFreshness() compares the RefreshState recorded at the last refresh against the
// *current* ViewVersion id, so a comment-only definition change incorrectly reports the view as STALE.
assertFreshness(materializedViewName, "STALE");
```
Commenting on a MV column - perfectly normal operation - has effect on the staleness check of the MV.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4061939516
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple path in the dependency graph.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its view metadata and one ore more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
Review Comment:
wrt `snapshot-id` being recorded for a source table.
If the source table is created and has no snapshots, is it supposed to be recorded or rather omitted in the source states?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4119676371
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
Review Comment:
IIUC `source-states` is a list which can be empty.
What benefit would it bring making it optional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4119721981
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+* **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+* **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple paths in the dependency graph.
+* **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+* **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+* **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+* **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
Review Comment:
> The engine consuming a materialized view
The process consuming a materialized view must not necessarily be an engine (e.g. pyiceberg).
I think that keeping the naming abstract to "Consumer" is a better wording fit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3967960238
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
Review Comment:
Iceberg snapshots internally contain the `schema-id` that was active at the time the snapshot was created. Therefore, to track schema evolution, engines do not need extra schema information injected into the refresh-state. During freshness evaluation, the query engine simply:
Looks at the source table snapshot-id recorded in the MV's refresh-state.
Checks the schema-id tied to that specific snapshot in the source table.
Compares it against the source table's current schema-id.
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
Review Comment:
Iceberg snapshots internally contain the `schema-id` that was active at the time the snapshot was created. Therefore, to track schema evolution, engines do not need extra schema information injected into the refresh-state. During freshness evaluation, the query engine simply:
- Looks at the source table snapshot-id recorded in the MV's refresh-state.
- Checks the schema-id tied to that specific snapshot in the source table.
- Compares it against the source table's current schema-id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4094144538
##########
format/view-spec.md:
##########
@@ -322,3 +453,201 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "catalog" : "prod",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+* `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+* `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+* `C` (regular view or materialized view, varies by strategy): `SELECT ... FROM F JOIN G ON ...`
Review Comment:
Good point — C and D are materialized views; what varies by strategy is how the producer records them (as tables in Strategy 2, expanded as views in Strategy 3, or omitted in Strategy 1), not their type. I've fixed the example to state they're materialized views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "osscm (via GitHub)" <gi...@apache.org>.
osscm commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4098811303
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+* **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+* **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple paths in the dependency graph.
+* **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+* **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+* **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
Review Comment:
Following up on this since it doesn't look like it made it into the spec text yet (I don't see anything about snapshot expiration/retention under source-table-state) - apologies if this was actually settled elsewhere and I missed it.
The scenario I keep getting stuck on: once a source table's retention/expiration policy removes the snapshot that was recorded in source-table-state.snapshot-id, the recorded reference just points at nothing anymore. At that point there's no way to compare "recorded snapshot" vs. "current snapshot" to decide whether the source actually changed - the comparison itself becomes undefined, not just "stale."
Is the intent that a missing/expired snapshot reference should always be treated as stale (safe, but can trigger unnecessary full refreshes on any table with a live expiration policy), or is there some other fallback in mind - e.g. keying off something more durable than the raw snapshot-id (the sequence-number idea mentioned earlier in this thread), or expecting producers to defensively re-record before expiration happens? Trying to understand whether "reference no longer resolves" is meant to be a distinct case from "reference resolves but doesn't match," or if I'm overthinking this.
cc @JanKaul @wmoustafa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4082661626
##########
format/view-spec.md:
##########
@@ -160,7 +179,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+* **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+* **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple path in the dependency graph.
+* **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+* **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+* **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+* **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its view metadata and one ore more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `catalog` | A string specifying the name of the catalog. |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
+| _optional_ | `ref` | Branch name of the source table being referenced in the view query |
+
+When `ref` is `null` or not set, it defaults to `main`.
Review Comment:
I think the `FOR VERSION AS OF` case mostly makes sense for fixed snapshot-ids that have a ref. Because otherwise you don't know how long the snapshot will exist or if it gets removed by compaction. And in such cases you can specify the ref in the `refresh-state` and should be able to determine the freshness correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3320715517
##########
format/view-spec.md:
##########
@@ -322,3 +449,215 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+- `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+- `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+- `C` (regular view or materialized view, varies by strategy): `SELECT ... FROM F JOIN G ON ...`
+- `D` (regular view or materialized view, varies by strategy): `SELECT ... FROM H WHERE ...`
+- `E`, `F`, `G`, `H`: base Iceberg tables
+
+### Strategy 1: Track all nested tables and views (no nested MVs)
+
+The view query reads only base tables and regular views. The refresh state tracks snapshot IDs of all deeply nested base tables and version IDs of all views traversed. Reuse of the storage table is sensitive to changes in any of them.
+
+`C` and `D` are regular views.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [VIEW] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [VIEW] <-- recorded in A: version-id: 7
+ ├── F [TABLE] <-- recorded in A: snapshot-id: 102
+ └── G [TABLE] <-- recorded in A: snapshot-id: 103
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query reads from materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is **not** recorded.
+
+`C` and `D` are materialized views, treated as views (expanded).
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV — expanded as view] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [MV — expanded as view] <-- recorded in A: version-id: 7
Review Comment:
This example would be more interesting if C's storage table was stale. Otherwise, the refresh state here is the same as the refresh state from Strategy 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3320636746
##########
format/view-spec.md:
##########
@@ -322,3 +449,215 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+- `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+- `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+- `C` (regular view or materialized view, varies by strategy): `SELECT ... FROM F JOIN G ON ...`
+- `D` (regular view or materialized view, varies by strategy): `SELECT ... FROM H WHERE ...`
+- `E`, `F`, `G`, `H`: base Iceberg tables
+
+### Strategy 1: Track all nested tables and views (no nested MVs)
+
+The view query reads only base tables and regular views. The refresh state tracks snapshot IDs of all deeply nested base tables and version IDs of all views traversed. Reuse of the storage table is sensitive to changes in any of them.
+
+`C` and `D` are regular views.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [VIEW] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [VIEW] <-- recorded in A: version-id: 7
+ ├── F [TABLE] <-- recorded in A: snapshot-id: 102
+ └── G [TABLE] <-- recorded in A: snapshot-id: 103
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query reads from materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is **not** recorded.
+
+`C` and `D` are materialized views, treated as views (expanded).
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV — expanded as view] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [MV — expanded as view] <-- recorded in A: version-id: 7
Review Comment:
This example would be more interesting if the storage table for C was stale so that this example didn't have the same refresh state as Strategy 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3397312060
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an upstream materialized view's storage table) and/or a `view` entry. Upstream materialized views can be stored as a `view` and a `table` entry.
Review Comment:
my earlier comment about the term `source-table` is also problematic here. it would imply a table entry is only a directly referenced table or MV's storage table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4149604359
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
Review Comment:
Agree with @JanKaul. We do not need to start introducing new concepts or explain beyond what is strictly required. I don't think this implies anything about refresh or serving, so we don't need to clarify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1741477865
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
+
+For that the additional field "refresh-state" is introduced as an opaque record in the table snapshot summary.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string. |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view. It has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#soure-table) records |
Review Comment:
It actually mentions the fully expanded query tree just above in line 201. But I can also add it to the description of the fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1742350128
##########
format/view-spec.md:
##########
@@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
+The storage table can have the states "fresh", "stale" or "invalid".
Review Comment:
maybe a minor change from `can have` to `can be` would avoid the misinterpretation. `have` can imply `containing the state field`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1851160036
##########
format/view-spec.md:
##########
@@ -158,6 +175,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Partial identifier
+
+The partial identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
Review Comment:
I would simplify this. ( no need to state 'additional metadata' or 'introduced' as its obvious from the context)?
```A snapshot property called 'refresh state' in the storage table determines the freshness of the precomputed data in that table. ```
Can we also link to snapshot property section of the table spec?
##########
format/view-spec.md:
##########
@@ -81,9 +94,13 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [partial identifier](#partial-identifier) of the storage table |
Review Comment:
Nit: align the column with previous line?
##########
format/view-spec.md:
##########
@@ -158,6 +175,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Partial identifier
+
+The partial identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
Review Comment:
Sorry if I forgot the discussion about this, but why not have UUID here as well? As its UUID for the source table reference.
##########
format/view-spec.md:
##########
@@ -158,6 +175,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Partial identifier
+
+The partial identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
+
+For that the additional field "refresh-state" is introduced as an opaque record in the table snapshot summary.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. It has the following fields:
Review Comment:
Can we define 'indirect references'?
##########
format/view-spec.md:
##########
@@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
+The storage table can be in the states of "fresh", "stale" or "invalid".
Review Comment:
I also think we should explicitly explain these states, if we mention them.
##########
format/view-spec.md:
##########
@@ -42,12 +42,25 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
Review Comment:
Its a bit of repeated information from down below. If this is more of an intro, high level paragraph, how about just
```Materialized views maintain additional metadata to track its storage table and its freshness.```
##########
format/view-spec.md:
##########
@@ -42,12 +42,25 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
Review Comment:
Sounds a bit over complicated to mention `option to turn into`, why not just
``` Materialized views are a type of view that precompute the data from the view query```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1876688205
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the views' catalog |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+A property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. Indirect references are the tables/views that are not directly referenced in the query but are nested within other views. The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#source-table) records for all tables that are directly or indirectly referenced in the materialized view query |
+| _required_ | `source-view-states` | A list of [source view](#source-view) records for all views that are directly or indirectly referenced in the materialized view query |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source table
+
+A source table record captures the state of a source table at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
Review Comment:
If i understand correctly @wmoustafa comment on the mailing list, then there is some ambiguity here for what to put, if the same table in expressed in the various forms (catalog.database.name) or (database.name) or (name), either in same sql statement or in the different sql representations of the same view. @stevenzwu i wonder do we dictate a standard to follow in this case?
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view that precompute the data from the view query.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`'s of the last refresh operation match the current `snapshot_id`'s of the source tables.
Review Comment:
Nit: i think we shouldnt have the apostrophe here (it seems a possessive one then of `snapshot_id`, which is wrong). Same for the next one.
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the views' catalog |
Review Comment:
should this be `view's` catalog, as view is singular here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1868293537
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view that precompute the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
Review Comment:
```suggestion
When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1870193131
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the views' catalog |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+A property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. Indirect references are the tables/views that are not directly referenced in the query but are nested within other views. The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#source-table) records for all tables that are directly or indirectly referenced in the materialized view query |
+| _required_ | `source-view-states` | A list of [source view](#source-view) records for all views that are directly or indirectly referenced in the materialized view query |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source table
+
+A source table record captures the state of a source table at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
Review Comment:
As I said on the mailing list, I don't have a strong opinion about this issue. I was hesitant to add it because I wanted to resolve the discussion with Walaa first. I think he might be skeptical about including the table identifier here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1958702676
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Table Identifier
Review Comment:
nit: let's call it `Storage table identifier` and update the reference link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2085450857
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
Review Comment:
```suggestion
Materialized View metadata is a superset of View metadata with an additional pointer to the storage table and refresh metadata.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "hashhar (via GitHub)" <gi...@apache.org>.
hashhar commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2048365318
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialzied views.
+The property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. Indirect references are the tables/views that are not directly referenced in the query but are nested within other views. The refresh state has the following fields:
Review Comment:
for finding indirect references how would implementors do that? For views I don't think there's explicit tracking of dependencies in the metadata. (I'm new to working with the spec, so not caught up fully with all the details - forgive for any stupid questions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2143429478
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table and refresh metadata.
+Refresh metadata contains information about the "source tables", tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of the source tables.
+* **stale** -- The `snapshot_id`s do not match, indicating that a refresh operation needs to be performed to capture the latest source table changes.
+* **invalid** -- The current `version_id` of the materialized view does not match the `refresh-version-id` of the refresh state.
+
## Specification
### Terms
* **Schema** -- Names and types of fields in a view.
* **Version** -- The state of a view at some point in time.
+* **Storage table** -- Iceberg table that stores the precomputed data of the materialized view.
+* **Source table** -- A table reference that occurs in the query definition of the materialized view. The materialized view depends on the data from the source tables.
+* **Source view** -- A view reference that occurs in the query definition of the materialized view. The materialized view depends on the definitions from the source views.
Review Comment:
MV SQL can reference tables or views. They are called source tables or views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2167178851
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
Review Comment:
@JanKaul can you add the link from table to this MV storage table section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2587876411
##########
format/view-spec.md:
##########
@@ -63,11 +79,13 @@ The view version metadata file has the following fields:
| _required_ | `versions` | A list of known [versions](#versions) of the view [1] |
| _required_ | `version-log` | A list of [version log](#version-log) entries with the timestamp and `version-id` for every change to `current-version-id` |
| _optional_ | `properties` | A string to string map of view properties [2] |
+| _optional_ | `max-staleness-ms` | The maximum time interval in milliseconds after a refresh operation during which the materialized view's data is considered fresh [3] |
Notes:
1. The number of versions to retain is controlled by the view property: `version.history.num-entries`.
2. Properties are used for metadata such as `comment` and for settings that affect view maintenance. This is not intended to be used for arbitrary metadata.
+3. The `max-staleness-ms` field only applies to materialized views and must be set to `null` for common views. If `max-staleness-ms` is not `null` and the time elapsed since the last refresh operation is less than `max-staleness-ms`, the query engine may return data directly from the `storage-table` without evaluating freshness based on the source tables and views. If `max-staleness-ms` is `null` for a materialized view, the data in the `storage-table` is always considered fresh.
Review Comment:
Thanks for your explanation. I've updated the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2590268273
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of all the source tables, OR all source table snapshots that differ from the last refresh have timestamps within a configured staleness window.
+* **stale** -- The `snapshot_id`s do not match for at least one source table and at least one differing snapshot has a timestamp outside the configured staleness window, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
> at least one source table and at least one differing snapshot
let's emphasize the word *and*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2594717873
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of all the source tables, OR all source table snapshots that differ from the last refresh have timestamps within a configured staleness window.
+* **stale** -- The `snapshot_id`s do not match for at least one source table and at least one differing snapshot has a timestamp outside the configured staleness window, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
Thanks you very much for taking improving my wording. I've updated the text accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2474661166
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
Review Comment:
Since there are many use cases for allowing an engine to use a stale materialization, can we add a "warm" situation with the description:
**warm** - The `snapshot_id`s do not match for at-least one source table and the snapshot was committed after the current time minus materialization.max-stalessness-ms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2474900955
##########
format/view-spec.md:
##########
@@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
I initially thought of `max-staleness` as a configuration property more than a metadata field. I think we touched on this in the last sync but didn't come to a conclusion.
Anyway, I like your description and I can add it either as a property or a metadata field, if there is consensus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2627854951
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
+
+Different systems interpret freshness differently, typically based on the objects referenced in the fully expanded query tree of the materialized view. Some systems consider only direct children, others only leaf nodes, and some the entire query tree. The specific interpretation is determined by the producer of the storage table.
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time. It contains a list of directly or indirectly referenced source states that allow a consumer to determine the freshness of the materialized view.
Review Comment:
Suggestion:
The refresh state record captures the **unique dependencies in the materialized view's dependency graph**. These dependencies include source Iceberg tables, views, and **nested** materialized views that allow a consumer to determine the freshness of the materialized view.
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
+
+Different systems interpret freshness differently, typically based on the objects referenced in the fully expanded query tree of the materialized view. Some systems consider only direct children, others only leaf nodes, and some the entire query tree. The specific interpretation is determined by the producer of the storage table.
Review Comment:
Suggestion:
Different systems define freshness differently, **based on how much of the dependency graph must be current**. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, “fresh” can mean strict end-to-end consistency, acceptable lag, or policy/version compliance.
##########
format/view-spec.md:
##########
@@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+
## Specification
### Terms
* **Schema** -- Names and types of fields in a view.
* **Version** -- The state of a view at some point in time.
+* **Storage table** -- Iceberg table that stores the precomputed data of a materialized view.
+* **Source table** -- A table reference that occurs in the query definition of a materialized view. The materialized view depends on the data from the source tables.
+* **Source view** -- A view reference that occurs in the query definition of a materialized view. The materialized view depends on the definitions from the source views.
Review Comment:
Suggestion:
Add this additional term:
**Nested materialized view** -- A dependent materialized view that is used in refreshing the current materialized view.
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
Review Comment:
Suggestion:
A materialized view is considered fresh when its precomputed data is usable by consumers. As tables **and views** referenced by a materialized view change over time, the precomputed data may no longer accurately **reflect the materialized view's dependency graph**. When this occurs, the materialized view (storage table) is considered stale.
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
+
+Different systems interpret freshness differently, typically based on the objects referenced in the fully expanded query tree of the materialized view. Some systems consider only direct children, others only leaf nodes, and some the entire query tree. The specific interpretation is determined by the producer of the storage table.
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time. It contains a list of directly or indirectly referenced source states that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+
+**Consumer evaluation:**
+- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`.
+- The consumer may additionally compare the `source-states` list against the states loaded from the catalog.
+- The consumer trusts that the producer has provided all states necessary to determine freshness.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Materialized views can reference source objects of different types, such as Iceberg tables and views. Source state records have a common field `type` that determines the form, which can be one of the following:
+
+* `table`: An Iceberg table
+* `view`: An Iceberg view
Review Comment:
Discussion:
Could we make it easier for the consumer to know whether a producer used a nested MV or not? The producer could separate out the nested MV with an additional type:
- materialized-view: An Iceberg materialized view used to refresh the current storage table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2629767433
##########
format/view-spec.md:
##########
@@ -160,6 +177,89 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is considered fresh when its precomputed data is usable by consumers. As tables referenced by a materialized view change over time, the precomputed data may no longer accurately reflect the logical materialized view definition. When this occurs, the materialized view (storage table) is considered stale.
Review Comment:
> Suggestion:
>
> A materialized view is considered fresh when its precomputed data is usable by consumers. As tables **and views** referenced by a materialized view change over time, the precomputed data may no longer accurately **reflect the materialized view's dependency graph**. When this occurs, the materialized view (storage table) is considered stale.
For me "reflect the materialized view's dependency graph" doesn't have the same meaning. In many cases the graph will be the same just that the data at the nodes changed. In my opinion that's not clear in your formulation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2791139552
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
Review Comment:
this reads a bit too strict almost like:
"must store records if conflicts present and conflict resolution must follow the rules ..."
( which is a bit in conflict with the choice not to store things at all)
may be rephrase sentence as:
" if a stored entry object is reachable through multiple paths in the dependency graph (diamond dependency pattern) then must store the entry with the oldest snapshot-id"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2791922584
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
Review Comment:
I don't exactly know what you mean. Does one of the following interpretations fit you idea?
1. Introduce a "Refresh state" reference below "Storage table metadata" and above "Freshness" and keep the "Refresh state" section below
2. Move the entire "Refresh state" section below the "Storage table metadata" and above the "Freshness" section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2807755244
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
makes sense (we should probably spell it out in the spec)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2797146438
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the dependency graph must be current.
+Some require the entire query tree to be fully up to date, while others only require direct children or leaf nodes.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
+- May leave the source states list empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+- Must store the entry with the oldest snapshot-id or version-id when the same source object is reachable through multiple paths in the dependency graph (diamond dependency pattern).
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and nested materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of nested materialized views |
+| `view` | An Iceberg view, including nested materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
Review Comment:
I think the binding is not strictly required as all entities have to be fetched from the catalog to get their snapshot-ids/version-ids. At that point the binding is clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3335419014
##########
format/view-spec.md:
##########
@@ -322,3 +449,215 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+- `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+- `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+- `C` (regular view or materialized view, varies by strategy): `SELECT ... FROM F JOIN G ON ...`
+- `D` (regular view or materialized view, varies by strategy): `SELECT ... FROM H WHERE ...`
+- `E`, `F`, `G`, `H`: base Iceberg tables
+
+### Strategy 1: Track all nested tables and views (no nested MVs)
+
+The view query reads only base tables and regular views. The refresh state tracks snapshot IDs of all deeply nested base tables and version IDs of all views traversed. Reuse of the storage table is sensitive to changes in any of them.
+
+`C` and `D` are regular views.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [VIEW] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [VIEW] <-- recorded in A: version-id: 7
+ ├── F [TABLE] <-- recorded in A: snapshot-id: 102
+ └── G [TABLE] <-- recorded in A: snapshot-id: 103
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query reads from materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is **not** recorded.
+
+`C` and `D` are materialized views, treated as views (expanded).
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV — expanded as view] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [MV — expanded as view] <-- recorded in A: version-id: 7
Review Comment:
I think the main point here is the semantics of recursively expanding the dependencies just like it were a view. I think the whole point is that the MV is treated like a view, so it is naturally similar to Strategy. I don' think that's an issue. The difference is that Strategy 1 are common views, while here it is materialized views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242602282
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+A snapshot whose refresh state violates a `Must` rule is invalid; consumers may treat it as if it had no `refresh-state`.
+
+##### Consumer options
+
+Consumers may use any combination of the following to assess the storage table:
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for intermediate materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table; otherwise it evaluates the view query in place of the storage table.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph. Each dependency is recorded in `source-states` as either a `table` entry (a base table or an intermediate materialized view's storage table) or a `view` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a base table in the dependency graph, or the storage table of an intermediate materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+An intermediate materialized view must be recorded as a single `table` entry referencing its storage table; recording it as a `view` entry is not permitted. The intermediate materialized view's own dependencies are reached recursively through its `refresh-state`.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
+| _optional_ | `ref` | Branch name of the source table being referenced in the view query |
+
+When `ref` is `null` or not set, it defaults to `main`.
+
+#### Source view state
+
+A source view record captures the state of a source view at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|--------------|-------------|
+| _required_ | `type` | A string that must be set to `view` |
+| _required_ | `name` | A string specifying the name of the source view |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
+| _required_ | `uuid` | The uuid of the source view |
+| _required_ | `version-id` | The version-id of the source view that was read during the refresh operation |
+
+#### Storage table creation and configuration
+
+When processing a `CREATE MATERIALIZED VIEW` statement, query engines must:
+
+1. Create the storage table as a regular Iceberg table with any specified configurations (partitioning, sort order, compression, etc.).
+2. Create the materialized view metadata with a `storage-table` reference pointing to the created storage table.
+
+The storage table must exist and be accessible before the materialized view metadata is committed.
Review Comment:
nit: `before` may be a bit strict. they can be committed in the same transaction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242422845
##########
format/view-spec.md:
##########
@@ -190,92 +190,93 @@ The table identifier for the storage table that stores the precomputed results.
### Storage table metadata
This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
-The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
| Requirement | Field name | Description |
|-------------|-----------------|-------------|
| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
#### Freshness
-A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
-Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
-**Consumer behavior:**
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
-When evaluating freshness, consumers:
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
-- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
-- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
-- May parse the view definition to implement more sophisticated policies.
-- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
-- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+##### Producer flexibility
-**Producer behavior:**
-
-Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
-Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
-Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
When writing the refresh state, producers:
-- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
-- If the source state cannot be determined for all objects (for example, for non-Iceberg tables or non-deterministic functions) may leave the source states list empty.
-- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+A snapshot whose refresh state violates a `Must` rule is invalid; consumers may treat it as if it had no `refresh-state`.
+
+##### Consumer options
+
+Consumers may use any combination of the following to assess the storage table:
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for intermediate materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table; otherwise it evaluates the view query in place of the storage table.
#### Refresh state
-The refresh state record captures the dependencies in the materialized view's dependency graph.
-These dependencies include source Iceberg tables, views, and materialized views.
+The refresh state record captures the dependencies in the materialized view's dependency graph. Each dependency is recorded in `source-states` as either a `table` entry (a base table or an intermediate materialized view's storage table) or a `view` entry.
Review Comment:
same as above let's use the same term as tables "base"; intermediate adds confusion IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242645848
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
Review Comment:
```suggestion
##### Recording Refresh State
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2882312436
##########
format/view-spec.md:
##########
@@ -322,3 +453,96 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```
Review Comment:
I wanted to keep the same style as the view examle. And I don't want to change the view example as this would be out of scope for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2882264160
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3033372305
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
Review Comment:
+1 to additional metadata to help the consumer with more determinism @bennychow.
By the way the first section `1. What is logically a sufficient refresh state` is just for reference and not for enforcement. I think we should state it to eliminate ambiguity. Sections 2 and 3 still give the flexibility we discussed.
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized views' |
Review Comment:
There is some resolution problem when storing the child table identifiers on the storage table side. This would be addressed if fully-resolved source table identifiers are recorded on the view version itself, as part of the view metadata.
#### Example illustrating the problem
Consider a materialized view with this query:
```sql
SELECT a.id, b.value FROM N.A JOIN N.B ON a.id = b.id
```
Created with `default-catalog = "C"` and `default-namespace = ["default"]`.
The refresh state in the storage table's snapshot summary records:
```json
{
"source-states": [
{"type": "table", "name": "A", "namespace": ["N"]},
{"type": "table", "name": "B", "namespace": ["N"]}
]
}
```
These entries have `namespace + name` but no catalog. A consumer reading the storage table's snapshot cannot resolve `A` and `B` to actual tables without knowing the catalog. One might assume the catalog can be obtained from the view's `default-catalog`, but this doesn't hold in the general case: if the MV's query references an intermediate view, and that view's SQL references tables under a different `default-catalog`, the source entries from the expanded view have no link back to the intermediate view whose `default-catalog` resolved them.
In short: source identifiers in the refresh state are incomplete, and the missing part (catalog) is not recoverable.
#### Proposed fix
The producer has the full resolution context at refresh time: it knows the catalog, namespace, and name of every source table after SQL analysis. If the producer records fully-resolved source identifiers on the view version, the resolution problem disappears:
```json
{
"version-id": 1,
"default-catalog": "C",
"default-namespace": ["default"],
"storage-table": {"namespace": ["default"], "name": "my_mv__storage"},
"source-tables": [
{"catalog": "C", "namespace": ["N"], "name": "A"},
{"catalog": "C", "namespace": ["N"], "name": "B"}
]
}
```
Each source entry is now independently resolvable — `C.N.A` and `C.N.B`, with no dependency on `default-catalog`.
#### Added benefit: state completeness without parsing
With source identifiers on the view version, a consumer can compare the view's declared `source-tables` against the refresh state's `source-states` to determine whether the producer recorded complete state without parsing the SQL. If the view declares sources `[A, B]` and the refresh state only contains `[A]`, the consumer knows the state is partial and can treat the MV as stale if it prefers.
This cleanly separates concerns:
- **View version** records the structural dependency: what tables the MV depends on (fully-resolved identifiers).
- **Refresh state** (in the storage table snapshot summary) records the temporal state: what snapshot each source was at when the refresh ran.
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
@stevenzwu We did have cross-catalog child tables at LinkedIn, so this is a pattern I've seen in practice. That said, I agree it's cleaner to avoid it. Constraining child tables to the same catalog as the view would significantly simplify the spec, particularly around identifier resolution and freshness checking. I'd be supportive of that constraint. The limitation is small and not worth the complexity it introduces.
**Worth noting:** this constraint doesn't prevent querying across data sources. A single catalog can still federate access to BigQuery, Snowflake, or other systems. The catalog is just an alias; same-catalog doesn't mean same-storage. @talatuyarer FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3033357480
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
Review Comment:
Case C is what I think we agreed on (during 4/2's sync) to be the default behavior in Spark. Case B is really an optimization of Case C since the consumer could recursively fetch the refresh state for the MV tables and combine them together.
I think we are trying to improve the case where consumer doesn't parse SQL and has enough information to decide whether it can use the MV's storage table or not.
The producer could include the following information into the refresh state to help this consumer:
- used-source-mvs: boolean - Whether the refresh used source materialized view(s).
- expanded-source-mvs: boolean - Whether the refresh state expanded source materialized view(s). (i.e. case B or case C)
- contains-non-iceberg-sources: boolean - Whether the materialized view's SQL included non-Iceberg sources
This information is easy for the producer to write into the refresh state and helps convey freshness intent. A non-SQL parsing consumer could have a requirement that it can only use MVs that refresh with other MVs as long as it can deeply validate all dependencies (i.e. used-source-mvs=true, expanded-source-mvs=true and contains-non-iceberg-tables=false).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3440784381
##########
format/view-spec.md:
##########
@@ -42,12 +42,27 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables", "source views", and/or "source materialized views", which are the tables/views/materialized views referenced in the query definition of the materialized view.
+
## Specification
### Terms
* **Schema** -- Names and types of fields in a view.
* **Version** -- The state of a view at some point in time.
+* **Storage table** -- Iceberg table that stores the precomputed data of a materialized view.
+* **Refresh state** -- A record stored in the storage table's snapshot summary that captures the state of source tables and views at the time of the last refresh operation.
+* **Dependency graph** -- The graph of all source tables, views, and materialized views that a materialized view depends on, including nested dependencies.
+* **Source table** -- A table reference that occurs in the query definition of a materialized view.
Review Comment:
Thanks Steven for catching this.
> How about simply: A table reference that appears in the dependency graph.
I thought we wanted to contain the use of dependency to the "Refresh-state" section.
I went with the formulation that we used in the last sync. Check, if it works for you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3012454968
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
+| _optional_ | `ref` | Branch name of the source table being referenced in the view query |
+
+When `ref` is `null` or not set, it defaults to "main".
+
+#### Source view state
+
+A source view record captures the state of a source view at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `view` |
+| _required_ | `name` | A string specifying the name of the source view |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If set to `null` the catalog is the same as the materialized views' |
+| _required_ | `uuid` | The uuid of the source view |
+| _required_ | `version-id` | The version-id of the source view that was read during the refresh operation |
+
+#### Storage table creation and configuration
+
+When processing a `CREATE MATERIALIZED VIEW` statement, query engines must:
+
+1. Create the storage table as a regular Iceberg table with any specified configurations (partitioning, sort order, compression, etc.).
Review Comment:
> This is also not a query engine requirement as the catalog implementation should handle this (not the engine directly).
if the creation of storage table is handled by the catalog, where does the client engine pass in the storage table config? do we need to introduce a `materialized-view` endpoint for create MV?
Engines define the DDL syntax `CREATE MATERIALIZED VIEW ...` for storage table configs like sort order, partitioning etc. It seems reasonable that engines generate the full metadata for storage table creation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3014391439
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
Review Comment:
how about:
"... adequately represents the view query at the current state of its dependencies."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3014391439
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
Review Comment:
how about:
"... adequately represents the result of the view query at the current state of its dependencies."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3010742795
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
Review Comment:
> property of every storage table snapshot
I don't think we need to state that it must be on _every_ table snapshot. That is inconsistent with other cases where we use snapshots to capture state (e.g. kafka offsets, checkpoints). The state should be set on the specific snapshot where it was captured. If there are additional snapshots (like maintenance operations), they do not need to create the state entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3968769731
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
Review Comment:
> The statement, while not wrong, leaves place for interpretations and doubt.
The current formulation was deliberately chosen to leave room for interpretations. We made multiple attempts to define "freshness" for all query engines and discarded it all together.
The goal is to provide all the necessary information of consumers in the `refresh-state` but not enforce anything. I think your strategy for the schema drift plucks a real whole and I will add it to the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3967970001
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple path in the dependency graph.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its view metadata and one ore more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
Review Comment:
Iceberg snapshots internally contain the `schema-id` that was active at the time the snapshot was created. Therefore, to track schema evolution, engines do not need extra schema information injected into the refresh-state. During freshness evaluation, the query engine simply:
- Looks at the source table snapshot-id recorded in the MV's refresh-state.
- Checks the schema-id tied to that specific snapshot in the source table.
- Compares it against the source table's current schema-id.
If the schema IDs differ (e.g., a column was renamed or a default value was added), the engine immediately knows the MV is semantically stale or requires validation, even if the underlying data files haven't changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4122229952
##########
format/view-spec.md:
##########
@@ -322,3 +454,201 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "catalog" : "prod",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+* `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+* `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+* `C` (materialized view): `SELECT ... FROM F JOIN G ON ...`
+* `D` (materialized view): `SELECT ... FROM H WHERE ...`
+* `E`, `F`, `G`, `H`: source Iceberg tables
+
+### Strategy 1: Empty refresh state (recency only)
+
+The producer leaves `source-states` empty and relies entirely on `refresh-start-timestamp-ms`. Consumers reuse the storage table based on a recency policy alone.
+
+`A`'s refresh state:
+
+```json
+{
+ "view-version-id": 1,
+ "refresh-start-timestamp-ms": 1573518435000,
+ "source-states": []
+}
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID and view version ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14, version-id: 9
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12, version-id: 7
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query reads from materialized views. The producer treats each materialized view as a regular view: expand through the MV's view definition and record the underlying tables and views. The MV's storage table snapshot ID is **not** recorded.
+
+`C` and `D` are materialized views, treated as views (expanded).
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV — expanded as view] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [MV — expanded as view] <-- recorded in A: version-id: 7
+ ├── F [TABLE] <-- recorded in A: snapshot-id: 102
+ └── G [TABLE] <-- recorded in A: snapshot-id: 103
+```
+
+The recorded shape matches Strategy 1. The difference is semantic: `C` and `D` are materialized views whose view definitions were expanded; their storage tables are not part of the recorded state.
+
+### Strategy 4: Track only view versions
+
+The producer treats the storage table as reusable as long as the view definitions in the dependency chain are unchanged. Underlying table changes do not affect freshness. Only view version IDs are recorded.
+
+`C` and `D` are regular views.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] (not recorded)
+│ └── D [VIEW] <-- recorded in A: version-id: 9
+│ └── H [TABLE] (not recorded)
+└── C [VIEW] <-- recorded in A: version-id: 7
+ ├── F [TABLE] (not recorded)
+ └── G [TABLE] (not recorded)
+```
+
+Snapshots of `E`, `F`, `G`, `H` are not recorded. Reuse is sensitive to view-definition changes but insensitive to data changes in the underlying tables.
+
+### Strategy 5: Skip non-Iceberg dependencies
+
+The producer records only Iceberg sources and omits non-Iceberg dependencies entirely. Useful when the view query reads from a mix of Iceberg and non-Iceberg sources and the producer chooses to track only the Iceberg side.
+
+Assume the query reads from source Iceberg tables `E`, `F`, `G`, `H` and an additional non-Iceberg table `N`.
+
+```
+A [MV — being refreshed]
+├── E [TABLE] <-- recorded in A: snapshot-id: 101
+├── F [TABLE] <-- recorded in A: snapshot-id: 102
+├── G [TABLE] <-- recorded in A: snapshot-id: 103
+├── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── N [NON-ICEBERG TABLE] (omitted; not tracked)
+```
+
+`N` is omitted. Consumers cannot verify `N`'s state from the refresh state alone, but know that the data queried from `N` cannot be older than `refresh-start-timestamp-ms`.
Review Comment:
The difference between the current version and your suggestion is that the current version only applies to producers of the `refresh-state`. It only provides strategies on how to populate the entries.
Your suggestion introduces the term "serving policy" and applies to producers and consumers alike. To me it looks very similar to the discussion we had about "freshness interpretation", where different engines could also have different interpretations of. I'm not sure if introducing and **agreeing** on the term "serving policy" will go down the same route as the "freshness" term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4080052193
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
> On rollback to a prior version, what storage table is current?
@stevenzwu could you please sketch via SQL how to reach this scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "osscm (via GitHub)" <gi...@apache.org>.
osscm commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4099125197
##########
format/view-spec.md:
##########
@@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
Sorry, If I'm missing in the spec doc. Trying to understand Is there a way to know, what all fields are allowed to change and what all are not, when `ALTER ...` is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4119668210
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+* **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+* **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple paths in the dependency graph.
+* **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+* **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+* **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
Review Comment:
> Is the intent that a missing/expired snapshot reference should always be treated as stale (safe, but can trigger unnecessary full refreshes on any table with a live expiration policy)
@osscm let's talk with concrete SQL examples to avoid misunderstandings.
Asumming the MV is created with the following definition:
```
CREATE OR REPLACE mv AS SELECT * FROM T FOR VERSION AS OF 'snapshot-id';
```
Now let's say that in the meantime the snapshot denoted by `snapshot-id` does not exist anymore.
I'm assuming that that you want to know whether the MV is "stale" or "fresh" in such a situation.
The storage table result does not correspond anymore to the result of the query from the view.
In such a situation, the MV is "stale".
Let's also create separate granular threads for each question to avoid going in longer unproductive digressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2474212274
##########
format/view-spec.md:
##########
@@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
`max-staleness` is optional field
The maximum time-lag allowed for the view's data, in milliseconds. A query planner will consider the view 'stale' if its data is older than this duration. A value of 0 or null implies no staleness target is defined, and the view must be refreshed manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2612328840
##########
format/view-spec.md:
##########
@@ -160,6 +177,71 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time.
+
+* Source view states are stored in `source-view-states`. It includes indirect references — views nested within other views (excluding MVs).
Review Comment:
i think in the meeting we were converging on including MVs here. Lets revisit during next meeting to make sure there is no ambiguity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2616065681
##########
format/view-spec.md:
##########
@@ -160,6 +177,71 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of source tables, views, and materialized views at refresh time.
+
+* Source view states are stored in `source-view-states`. It includes indirect references — views nested within other views (excluding MVs).
+* Source table states are stored in `source-table-states`. It includes indirect references - tables nested within other views (excluding MVs).
+
+For directly referenced source materialized views, both the source view and its storage table are included in the refresh state. Indirect references (views or tables) from source materialized views are excluded in the refresh-state. During read time, a query engine recursively expands the query tree to determine freshness if it chooses to enforce recursive evaluation semantic.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#source-table) records for tables directly or indirectly referenced through common views, plus storage tables of directly referenced source materialized views |
+| _required_ | `source-view-states` | A list of [source view](#source-view) records for all views (including materialized views) that are directly referenced, plus common views indirectly referenced through other common views |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source table
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | Snapshot-id of when the last refresh operation was performed |
+| _optional_ | `ref` | Branch name of the source table being referenced in the view query |
+
+When `ref` is `null` or not set, it defaults to "main".
+
+#### Source view
+
+A source view record captures the state of a source view at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `uuid` | The uuid of the source view |
+| _required_ | `version-id` | Version-id of when the last refresh operation was performed |
+
+#### Status Interpretation
+
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+
+* **invalid** -- The current `version_id` of the materialized view does not match the `view-version-id` recorded in its refresh state. A read operation cannot proceed using the materialized view's data.
Review Comment:
I personally find trying to use invalid, fresh and stale statuses as too generic. We could instead describe the 3 high level ways a consumer would validate the freshness:
1. Use refresh-start-timestamp-ms
2. Use source-table-states and source source-view-states. Optionally validating nested MV refresh-state..
3. Advanced validation through view SQL expansion and refresh-state comparison.
For 3, the consumer may determine that affected partitions in the table are still up to date after pushing down filters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1779797796
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
Review Comment:
Yes, that's the motivation. It's a quick way to say that the materialization is at most current time minus refresh-start-timestamp-ms stale. It would also be useful if the view references non-Iceberg tables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1744844818
##########
format/view-spec.md:
##########
@@ -81,9 +93,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [full identifier](#full-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When 'storage-table' is `null` or not set, the entity is a common view, otherwise it is a materialized view.
+
Review Comment:
I didn't see any view or table properties described in the specification. That's why I haven't added any properties yet.
However, if you feel like we should, it's fine with me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1736801043
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
+
+For that the additional field "refresh-state" is introduced as an opaque record in the table snapshot summary.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string. |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view. It has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#soure-table) records |
Review Comment:
Also should we clarify that if the list contains the directly referenced tables only? or recursively referenced tables should be resolved to this list?
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
+
+For that the additional field "refresh-state" is introduced as an opaque record in the table snapshot summary.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string. |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view. It has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#soure-table) records |
Review Comment:
I have been thinking about if this should be a `list` or a `map`. I guess `list` is probably good since we don't really need lookup capability here. We need to iterate through the sources and compare the satate.
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
+
+For that the additional field "refresh-state" is introduced as an opaque record in the table snapshot summary.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string. |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view. It has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#soure-table) records |
+| _required_ | `source-view-states` | A list of [source view](#soure-view) records |
Review Comment:
I am wondering if this should be merged with the `source-table-states` as a single `source-states` for a list of referenced source tables and views. The only difference btw the two structs are the `id` part: `snapshot-id` vs `version-id`. I guess the benefit of separated list is to make state diff easier. it is a bit different logic to diff table state vs version state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1868938165
##########
format/view-spec.md:
##########
@@ -82,9 +98,13 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [partial identifier](#partial-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When 'storage-table' is `null` or not set, the entity is a common view, otherwise it is a materialized view.
+The catalog of the storage table must be same as the one of the materialized view.
Review Comment:
I've updated the document accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1869998317
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the views' catalog |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+A property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. Indirect references are the tables/views that are not directly referenced in the query but are nested within other views. The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#source-table) records for all tables that are directly or indirectly referenced in the materialized view query |
+| _required_ | `source-view-states` | A list of [source view](#source-view) records for all views that are directly or indirectly referenced in the materialized view query |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source table
+
+A source table record captures the state of a source table at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
Review Comment:
just want to call out a discussion point if we want to include the table identifier fields for readability
- catalog (optional)
- database (optional)
- table (required)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1885605556
##########
format/view-spec.md:
##########
@@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [full identifier](#full-identifier) of the storage table |
Review Comment:
I thought that a "Full Identifier" is a more general concept that could also be used in other places in the future. For example in the Refresh-state/lineage.
But if people would like to change the name, we can do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1885416865
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+A property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. Indirect references are the tables/views that are not directly referenced in the query but are nested within other views. The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
Review Comment:
maybe `view-verision-id`?
##########
format/view-spec.md:
##########
@@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [full identifier](#full-identifier) of the storage table |
Review Comment:
similar to the `partial` comment, maybe the description can just be `[Storage table identifier](storage-table-identifier)`
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the views' catalog |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+A property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. Indirect references are the tables/views that are not directly referenced in the query but are nested within other views. The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#source-table) records for all tables that are directly or indirectly referenced in the materialized view query |
+| _required_ | `source-view-states` | A list of [source view](#source-view) records for all views that are directly or indirectly referenced in the materialized view query |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source table
+
+A source table record captures the state of a source table at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
Review Comment:
This is not a blocker for me. Functionally, it is not required. I just thought the refresh state would be more human friendly if names (like database and name) are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1851160036
##########
format/view-spec.md:
##########
@@ -158,6 +175,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Partial identifier
+
+The partial identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
Review Comment:
I would simplify this. ( no need to state 'additional metadata' or 'introduced' as its obvious from the context)? Also not sure what is 'opaque record'?
```A snapshot property called 'refresh state' is set on the storage table to determine the freshness of the precomputed data in that table. ```
Can we also link to snapshot property section of the table spec?
##########
format/view-spec.md:
##########
@@ -158,6 +175,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Partial identifier
+
+The partial identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
Review Comment:
I would simplify this. ( no need to state 'additional metadata' or 'introduced' as its obvious from the context)? Also not sure what is 'opaque record'?
How about ```A snapshot property called 'refresh state' is set on the storage table to determine the freshness of the precomputed data in that table. ```
Can we also link to snapshot property section of the table spec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1741392915
##########
format/view-spec.md:
##########
@@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a fully resolved reference for a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `catalog` | A string specifying the catalog of the source table |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `table` | A string specifying the name of the source table |
+| _optional_ | `ref` | Branch name of the source table that is being referenced in the view query |
+
+When 'ref' is `null` or not set, it defaults to “main”. This field is to be ignored if the referenced entity is a view.
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+To be able to determine the freshness of the precomputed data, additional metadata is stored as part of the storage table.
+
+For that the additional field "refresh-state" is introduced as an opaque record in the table snapshot summary.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string. |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view. It has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `refresh-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-table-states` | A list of [source table](#soure-table) records |
Review Comment:
Yes, should clarify its recursively referenced tables and views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1910985763
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
Review Comment:
```suggestion
#### Table Identifier
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2631436927
##########
format/view-spec.md:
##########
@@ -160,6 +178,108 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view's precomputed data becomes stale as the tables and views referenced in its query definition change over time. Freshness determines whether the precomputed data accurately represents the logical query definition at the current state of its dependencies.
+
+Different systems define freshness differently, based on how much of the dependency graph must be current. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, "fresh" can mean strict end-to-end consistency, acceptable lag, or policy/version compliance.
+
+A materialized view is considered fresh when its precomputed data meets the freshness criteria defined by the consumer's evaluation policy. When these criteria are not met, the materialized view is considered stale.
+
+#### Refresh state
+
+The refresh state record captures the unique dependencies in the materialized view's dependency graph. These dependencies include source Iceberg tables, views, and nested materialized views that allow a consumer to determine the freshness of the materialized view.
Review Comment:
"unique dependencies" seem to indicate that either diamond pattern of queries is not supported or there is a policy to choose which instance of dependency to be selected if the same table queried multiple times ( indirectly through different upstream MVs at different points in time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3011229162
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
If we allow other catalogs, we need to store their names. And if we include the "catalog" name, we get the issue that different query engines might resolve it differently.
That's why we restrict it to the same catalog and avoid storing catalog identifiers in the metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3014525620
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation. If the producers interpretation is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), the entry with the oldest snapshot-id or version-id must be stored.
Review Comment:
With regards to freshness, to argument was as follows:
- all source tables need to be up-to-date for a storage table to be fresh
- a single out-of-date source table renders the storage table stale
- if any source table snapshot is out-of-date, the oldest snapshot is also out-of-date
- the oldest snapshot is enough information to determine freshness in the diamond pattern
Your comment is really relevant for incremental processing though. As the information what data has been processed is vital. I think we should then include all source table states, even different snapshots of the same table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3033123038
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
Review Comment:
I'd like to propose some simplificaition by decomposing this aspect to _three independent dimensions._ Specifically around:
* What constitutes sufficient state (logically)
* What flexibility producers have
* What options consumers have
### 1. What is logically a sufficient refresh state
The definition of "sufficient" depends on how intermediate materialized views are treated:
**Case A: No intermediate MVs (all sources are base tables, optionally through intermediate views)**
Sufficient state = snapshot IDs of all leaf tables referenced by the view's query, in addition to intermediate view version IDs. This is the straightforward case.
**Case B: Intermediate MVs treated as views (transparent expansion)**
The engine expands intermediate MVs into their underlying queries. This reduces to Case A — sufficient state is the snapshot IDs of all deep leaf tables, plus version IDs of all intermediate views traversed during expansion.
**Case C: Intermediate MVs treated as tables (opaque boundaries)**
The engine treats intermediate MVs as materialized data sources. Sufficient state = snapshot IDs of the intermediate MVs' storage tables. No expansion beyond the MV boundary. Freshness of the intermediate MV is that MV's own concern.
Without guidelines on what consitutes sufficiency, consumers have no way to distinguish "the producer intentionally recorded partial state" from "the producer recorded complete state." A consumer seeing two source entries has no signal whether that covers all sources or just the ones the producer chose to track. These definitions can be added to the spec for reference.
### 2. Producer options
Producers decide what state to record at refresh time. This can be partial — a producer may only track sources from the same catalog, may skip non-Iceberg sources, or may not expand intermediate MVs.
### 3. Consumer options
Consumers have two strategies:
- **Trust the recorded state.** Simple to implement, but only correct if the producer recorded complete state. No way to verify this from the refresh state alone.
- **Parse the view query independently.** The consumer re-analyzes the SQL, identifies all source tables, and checks whether the recorded state covers them. Correct but expensive — requires SQL parsing and cross engine dialect translation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3031253085
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
When `default-catalog` is `null` or not set, the catalog in which the view is stored must be used as the default catalog.
+When `storage-table` is `null` or not set, the entity is a common view, otherwise it is a materialized view. The storage table must be in the same catalog as the materialized view.
Review Comment:
Practically, are there use cases where storage tables won't be in the same catalog as the view definition?
If no, it will be great to keep this restriction. Conceptually, a single logical entity of MV consists of two parts: view and storage table. It is natural that both components live in the same catalog. [The new relation endpoint](https://docs.google.com/document/d/1VW5hgaaajRWtp5KbOU3s83YyoyPi5WOSvHtoJ_yXzJs/edit?tab=t.0#heading=h.e6w7vgpr8t2f) also work well with this restriction so that catalog can return the MV in one round-trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3033212803
##########
format/view-spec.md:
##########
@@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
+- If the source state cannot be determined for all objects (for example, for non-Iceberg tables) may leave the source states list empty.
+- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+
+#### Refresh state
+
+The refresh state record captures the dependencies in the materialized view's dependency graph.
+These dependencies include source Iceberg tables, views, and materialized views.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view including objects referenced by source materialized views.
+Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table, including storage tables of source materialized views |
+| `view` | An Iceberg view, including source materialized views |
+
+Source materialized views are represented by two source state entries: one for the view itself and one for its storage table.
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized views' |
Review Comment:
There is some resolution problem when storing the child table identifiers on the storage table side. This would be addressed if fully-resolved source table identifiers are recorded on the view version itself, as part of the view metadata.
#### Example illustrating the problem
Consider a materialized view with this query:
```sql
SELECT a.id, b.value FROM N.A JOIN N.B ON a.id = b.id
```
Created with `default-catalog = "C"` and `default-namespace = ["default"]`.
The refresh state in the storage table's snapshot summary records:
```json
{
"source-states": [
{"type": "table", "name": "A", "namespace": ["N"]},
{"type": "table", "name": "B", "namespace": ["N"]}
]
}
```
These entries have `namespace + name` but no catalog. A consumer reading the storage table's snapshot cannot resolve `A` and `B` to actual tables without knowing the catalog. One might assume the catalog can be obtained from the view's `default-catalog`, but this doesn't hold in the general case: if the MV's query references an intermediate view, and that view's SQL references tables under a different `default-catalog`, the source entries from the expanded view have no link back to the intermediate view whose `default-catalog` resolved them.
In short: source identifiers in the refresh state are incomplete, and the missing part (catalog) is not recoverable.
#### Proposed fix
The producer has the full resolution context at refresh time: it knows the catalog (from `default-catalog`, namespace, and name of every source table after SQL analysis.
```json
{
"version-id": 1,
"default-catalog": "C",
"default-namespace": ["default"],
"storage-table": {"namespace": ["default"], "name": "my_mv__storage"},
"source-tables": [
{"namespace": ["N"], "name": "A"},
{"namespace": ["N"], "name": "B"}
]
}
```
The partial identifiers `N.A` and `N.B` now co-exist with the `default-catalog` in the same metadata object (view), and can be jointly resolved.
#### Added benefit: state completeness without parsing
With source identifiers on the view version, a consumer can compare the view's declared `source-tables` against the refresh state's `source-states` to determine whether the producer recorded complete state without parsing the SQL. If the view declares sources `[A, B]` and the refresh state only contains `[A]`, the consumer knows the state is partial and can treat the MV as stale if it prefers.
This cleanly separates concerns:
- **View version** records the structural dependency: what tables the MV depends on (fully-resolved identifiers).
- **Refresh state** (in the storage table snapshot summary) records the temporal state: what snapshot each source was at when the refresh ran.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2783193184
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+Consumers should only read from the storage table if the materialized view is "fresh" and therefore adequately represents the logical query definition of the view.
+Different systems define freshness differently based on time-based and logical factors.
+
+**Time-based freshness (consumer-defined):**
+
+Consumers may apply time-based freshness policies, such as allowing a certain staleness window based on `refresh-start-timestamp-ms`.
+When evaluating freshness, consumers:
+- Must first evaluate their own time-based freshness policy.
+- May additionally compare the `source-states` list against the states loaded from the catalog to verify the producers logical freshness policy.
+- May parse the view definition to implement a more sophisticated policy.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Must not read from the storage table when the materialized view doesn't meet freshness criteria.
Review Comment:
"Must" here seem out of place We should have "may all bullet points here"
I think that was consensus in the room :
When ee say
1) Consumers may apply ...policies
2) when evaluating policies "must ...."
sounds strange
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2786379547
##########
format/view-spec.md:
##########
@@ -160,6 +176,109 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+Consumers should only read from the storage table if the materialized view is "fresh" and therefore adequately represents the logical query definition of the view.
+Different systems define freshness differently based on time-based and logical factors.
+
+**Time-based freshness (consumer-defined):**
+
+Consumers may apply time-based freshness policies, such as allowing a certain staleness window based on `refresh-start-timestamp-ms`.
+When evaluating freshness, consumers:
+- Must first evaluate their own time-based freshness policy.
+- May additionally compare the `source-states` list against the states loaded from the catalog to verify the producers logical freshness policy.
+- May parse the view definition to implement a more sophisticated policy.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Must not read from the storage table when the materialized view doesn't meet freshness criteria.
+
+**Logical freshness (producer-defined):**
Review Comment:
I rephrased it so that it only describes producer and consumer behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2807731497
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
> Is initially empty storage table would be easily distinguishable from recently refreshed but (happened to be) empty (as query retuned nothing) storage tables ?
This seems straightforward to me. Empty table after refresh would still have `refresh-state` populated with at least `last-refresh-time` even if the `source-state` is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2635378407
##########
format/view-spec.md:
##########
@@ -160,6 +178,108 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view's precomputed data becomes stale as the tables and views referenced in its query definition change over time. Freshness determines whether the precomputed data accurately represents the logical query definition at the current state of its dependencies.
+
+Different systems define freshness differently, based on how much of the dependency graph must be current. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, "fresh" can mean strict end-to-end consistency, acceptable lag, or policy/version compliance.
+
+A materialized view is considered fresh when its precomputed data meets the freshness criteria defined by the consumer's evaluation policy. When these criteria are not met, the materialized view is considered stale.
+
+#### Refresh state
+
+The refresh state record captures the unique dependencies in the materialized view's dependency graph. These dependencies include source Iceberg tables, views, and nested materialized views that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
+
+**Consumer evaluation:**
+- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`. A materialized view is fresh if `refresh-start-timestamp-ms` is within the window `[now - max-staleness-ms, now]`.
+- The consumer may additionally compare the `source-states` list against the states loaded from the catalog. If this evaluation determines the materialized view is fresh, it overrides the coarse-grained evaluation result.
+- The consumer may parse the view definition to implement a more sophisticated policy.
+- When a materialized view is considered stale, the consumer can fail, refresh inline, or treat the materialized view as a logical view. The consumer must not consume from the storage table when the materialized view is stale.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source states](#source-state) records |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+#### Source state
+
+Materialized views can reference source objects of different types, such as Iceberg tables, view, and materialized views. Source state records have a common field `type` that determines the form, which can be one of the following:
+
+* `table`: An Iceberg table
+* `view`: An Iceberg view
+* `materialized-view`: An Iceberg materialized view
+
+The metadata fields for each type are defined below:
+
+#### Source table state
+
+A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
Review Comment:
while it is simpler, I am wondering if it can be implemented correctly. How can it move forward? Without the lineage path, when a MV refresh detected a nested MV has been refreshed, how does the producer know that the older snapshot is from this lineage path or from another way? Without the knowledge, the producer won't know if it can move the state forward or not.
this discussion leads me to think about an orthogonal question. What if the snapshots referenced by the refresh-state have been purged due to retention? Do we need to spell out the consumer behavior in this case? Maybe fall back to the coarse-grained evaluation with `refresh-start-timestamp-ms` and `max-staleness-ms`, which is a required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2872925177
##########
format/spec.md:
##########
@@ -1846,3 +1846,7 @@ The Geometry and Geography class hierarchy and its Well-known text (WKT) and Wel
Points are always defined by the coordinates X, Y, Z (optional), and M (optional), in this order. X is the longitude/easting, Y is the latitude/northing, and Z is usually the height, or elevation. M is a fourth optional dimension, for example a linear reference value (e.g., highway milepost value), a timestamp, or some other value as defined by the CRS.
The version of the OGC standard first used here is 1.2.1, but future versions may also be used if the WKB representation remains wire-compatible.
+
+## Appendix H: Materialized Views
Review Comment:
nit: I don't see the "Appendix H: Materialized View" being referenced anywhere.
Also this seems to be the first reference made in this file towards `view-spec.md`.
What is the intention on having this appendix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2875625183
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+
+When writing the refresh state, producers:
+- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's interpretation.
Review Comment:
An empty line is missing after L219.
<img width="876" height="145" alt="Image" src="https://github.com/user-attachments/assets/458804a5-98b8-4a74-9b61-f55c0da62c02" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1958712529
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Table Identifier
+
+The table identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Materialized View Metadata stored as part of the Table Metadata
+
+A property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
Review Comment:
maybe add a clarifying sentence before the `refresh-state` sentence. e.g.
```
This section describes additional metadata for storage table that are required for materialized views.
```
I am also wondering if this should be added to the `Table spec`. we can add cross reference links btw the table and view specs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2746574923
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1964502576
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
Review Comment:
I am not too opinionated on this either. At least, I was wondering if this has been discussed and what does the community favor.
I remember from previous discussion @wmoustafa also favored all MV related info in one page. I can definitely understand why. We can keep it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2484438838
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2086100508
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of the source tables.
+* **stale** -- The `snapshot_id`s do not match, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
Thanks @chenjian2664 and @yyanyy for your comments. These are great points. The current definition supports nested views in that it fully expands the query tree at refresh time and stores all directly and indirectly referenced tables in the refresh-state. This can include multiple levels of views.
This currently would also apply to materialized views. However, I think you're making a good point. When referencing another materialized view it would be easier to just store the snapshot_id of its storage table instead of tracking all table references in the materialized view.
You're correct in assuming that `fresh`/`stale`/`invalid` are introduced to have some nomenclature, these to not occur in the metadata. I will try to make this clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2587813844
##########
format/view-spec.md:
##########
@@ -63,11 +79,13 @@ The view version metadata file has the following fields:
| _required_ | `versions` | A list of known [versions](#versions) of the view [1] |
| _required_ | `version-log` | A list of [version log](#version-log) entries with the timestamp and `version-id` for every change to `current-version-id` |
| _optional_ | `properties` | A string to string map of view properties [2] |
+| _optional_ | `max-staleness-ms` | The maximum time interval in milliseconds after a refresh operation during which the materialized view's data is considered fresh [3] |
Notes:
1. The number of versions to retain is controlled by the view property: `version.history.num-entries`.
2. Properties are used for metadata such as `comment` and for settings that affect view maintenance. This is not intended to be used for arbitrary metadata.
+3. The `max-staleness-ms` field only applies to materialized views and must be set to `null` for common views. If `max-staleness-ms` is not `null` and the time elapsed since the last refresh operation is less than `max-staleness-ms`, the query engine may return data directly from the `storage-table` without evaluating freshness based on the source tables and views. If `max-staleness-ms` is `null` for a materialized view, the data in the `storage-table` is always considered fresh.
Review Comment:
I may have misunderstood Dan's comment at: https://youtu.be/ZdUoeGOltkM?t=623. I thought that he was opposed to the delayed view semantics because some clients might not be able to determine the timestamps of the source table snapshots. But I think he was more referring to their capabilities to actually refresh the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2872969959
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is "fresh" when the storage table adequately represents the logical query definition of the view.
+Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+
+**Consumer behavior:**
+
+When evaluating freshness, consumers:
+- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
+- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
+- May parse the view definition to implement more sophisticated policies.
+- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
+- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+
+**Producer behavior:**
+
+Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
+Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
+Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
Review Comment:
> Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
Is this still actual after introducing the functionality https://github.com/apache/iceberg/pull/13810 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2872952820
##########
format/view-spec.md:
##########
@@ -322,3 +453,96 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
Review Comment:
With regards to repeated "refresh" operations, is there any note worth adding in the spec on expiring snapshots of the storage table.
Is the expiration of previous snapshots a query engine detail or there is an opportunity to add more details on the spec about how it is happening?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3318974280
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+A snapshot whose refresh state violates a `Must` rule is invalid; consumers may treat it as if it had no `refresh-state`.
+
+##### Consumer options
+
+Consumers may use any combination of the following to assess the storage table:
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
Review Comment:
NIT: "Recency" doesn't seem to be a common term used by the DBs ( commonly used terms are staleness , freshness ,lag ) . Unless we are trying to make point that is somehow a distinct feature I would prefer not to use an "uncommon" term here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3311802612
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
Review Comment:
Isn't that spelled out by the empty list in the sentence above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3335421764
##########
format/view-spec.md:
##########
@@ -322,3 +449,215 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+Producers may select different sets of dependencies to record in the refresh state. The strategies below illustrate common choices against the same shared query.
+
+### Shared query
+
+- `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+- `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+- `C` (regular view or materialized view, varies by strategy): `SELECT ... FROM F JOIN G ON ...`
+- `D` (regular view or materialized view, varies by strategy): `SELECT ... FROM H WHERE ...`
+- `E`, `F`, `G`, `H`: base Iceberg tables
+
+### Strategy 1: Track all nested tables and views (no nested MVs)
+
+The view query reads only base tables and regular views. The refresh state tracks snapshot IDs of all deeply nested base tables and version IDs of all views traversed. Reuse of the storage table is sensitive to changes in any of them.
+
+`C` and `D` are regular views.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [VIEW] <-- recorded in A: version-id: 9
+│ └── H [TABLE] <-- recorded in A: snapshot-id: 104
+└── C [VIEW] <-- recorded in A: version-id: 7
+ ├── F [TABLE] <-- recorded in A: snapshot-id: 102
+ └── G [TABLE] <-- recorded in A: snapshot-id: 103
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer stops at each MV boundary and records the MV's storage table snapshot ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table snapshot-id: 14
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's refresh-state)
+└── C [MV] <-- recorded in A: storage-table snapshot-id: 12
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C` and `D`'s dependency graphs.
Review Comment:
I agree, I've updated the document
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242318057
##########
format/view-spec.md:
##########
@@ -190,92 +190,93 @@ The table identifier for the storage table that stores the precomputed results.
### Storage table metadata
This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
-The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
| Requirement | Field name | Description |
|-------------|-----------------|-------------|
| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
#### Freshness
-A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies.
-Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy.
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
-**Consumer behavior:**
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
-When evaluating freshness, consumers:
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
-- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`.
-- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation.
-- May parse the view definition to implement more sophisticated policies.
-- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view.
-- Should not consume the storage table as it is when the materialized view doesn't meet the freshness criteria.
+##### Producer flexibility
-**Producer behavior:**
-
-Producers should provide the necessary information in the [refresh state](#refresh-state) such that consumers can verify the logical equivalence of the precomputed data with the query definition.
-Different producers may have different freshness interpretations, based on how much of the refresh state's dependency graph should be evaluated.
-Some producers expect the entire dependency graph to be evaluated and therefore include source MV dependencies. Other producers may only expect dependencies in the MV's SQL to be evaluated and therefore do not include dependencies of source MVs.
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
When writing the refresh state, producers:
-- Should provide a sufficient list of source states such that consumers can determine freshness according to the producer's intent. If the producers intent is such that it doesn't rely on the source-states to determine freshness, it may provide an empty list.
-- If the source state cannot be determined for all objects (for example, for non-Iceberg tables or non-deterministic functions) may leave the source states list empty.
-- If a stored object is reachable through multiple paths in the dependency graph (diamond dependency pattern), all distinct source states have to be included in the list.
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
Review Comment:
Must ....... they chose to track.
sounds awkward and we talk about sources that may tracked by produces but not consumers in the next line )
I think we can just use "Should" here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242372190
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
Review Comment:
this `must` seems a little weird to me. It seems to saying the same thing as "Producers may selectively choose a subset of their dependencies to record". I would be good to just remove this item.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242616779
##########
format/view-spec.md:
##########
@@ -322,3 +453,142 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: What counts as a dependency
+
+The dependencies of a materialized view are determined by parsing the view query:
+
+- **Base Iceberg tables** in the dependency graph are recorded by `snapshot-id`.
+- **Iceberg views** in the dependency graph are recorded by `version-id`. A view's own dependencies are transitively dependencies of the materialized view and appear as additional entries in `source-states`.
+- **Intermediate materialized views** in the dependency graph are treated as their storage tables and recorded by the storage table's `snapshot-id`. Their own freshness is established recursively from their `refresh-state`.
+
+### Example
+
+The query under examination:
+
+- `A` (the materialized view being refreshed): `SELECT ... FROM B JOIN C ON ...`
+- `B` (regular view): `SELECT ... FROM E JOIN D ON ...`
+- `C` (materialized view): `SELECT ... FROM F JOIN G ON ...`
+- `D` (materialized view): `SELECT ... FROM H WHERE ...`
+- `E`, `F`, `G`, `H`: base Iceberg tables
+
+`A`'s dependencies are `B`, `C`, and `D`. `B` is a regular view; its own dependencies (`E` and `D`) are transitively dependencies of `A`. `C` and `D` are materialized views; they appear in `A`'s `source-states` as their storage tables.
Review Comment:
why A depends on D here? D is not direct child of A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3242388776
##########
format/view-spec.md:
##########
@@ -160,7 +178,120 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query (at the materialized view's current `view-version-id`) over the current state of its dependencies. Dependencies are determined by parsing the SQL: base Iceberg tables, Iceberg views (whose own dependencies are transitively dependencies of the materialized view), and intermediate materialized views (treated as their storage tables, with their own freshness established recursively from their `refresh-state`).
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is not fresh under the current definition.
+
+The `refresh-state` summary on each storage-table snapshot records dependency state observed at refresh time. Producers populate it; consumers use it to assess freshness without re-executing the query. The spec does not mandate what producers record or how consumers assess. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for what counts as a dependency.
+
+##### Producer flexibility
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Must** include all distinct source states for the inputs they chose to track.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+A snapshot whose refresh state violates a `Must` rule is invalid; consumers may treat it as if it had no `refresh-state`.
Review Comment:
If the producer is not spec compliant, should consumers treat the MV as invalid and not consumable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3968684870
##########
format/view-spec.md:
##########
@@ -160,7 +178,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+- **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+- **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple path in the dependency graph.
+- **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+- **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+- **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+- **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
+
+#### Source state
+
+Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form:
+
+| Type | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view |
+| `view` | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its view metadata and one ore more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies.
+
+#### Source table state
+
+A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name | Description |
+|-------------|---------------|-------------|
+| _required_ | `type` | A string that must be set to `table` |
+| _required_ | `name` | A string specifying the name of the source table |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _optional_ | `catalog` | An optional name of the catalog. If not set, the catalog is the same as the materialized view's |
+| _required_ | `uuid` | The uuid of the source table |
+| _required_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation |
Review Comment:
I don't think we have covered that scenario, thanks for bringing it up. The reason for not using the `metadata-file-location` was that iceberg generally wants to move the REST catalog and potentially deprecate the metadata.json file at some point.
Yeah, I think your strategy for detecting schema drift should work. We should probably add it to the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "osscm (via GitHub)" <gi...@apache.org>.
osscm commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4098902519
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
>A new view version can carry a different storage-table identifier than the previous version. Is that allowed?
I believe, we should not allow to do this operation, keep it immutable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4073475523
##########
format/view-spec.md:
##########
@@ -322,3 +449,198 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
Review Comment:
Per spec definition though
> A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
The alternative is to simply carry forward the refresh-state in a new snapshot of the storage table when updating the view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4099537122
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+* **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+* **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple paths in the dependency graph.
+* **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
Review Comment:
Reading this again, I think it could sound a bit confusing due to: Must, Should, May, as well as mention of some optional strategies that are better suited to Appendix B. What do you think about this?
```suggestion
The engine-specific materialized view's serving policy determines which entries a producer or engine records in `source-states`. At a minimum, a producer must record `view-version-id` and `refresh-start-timestamp-ms`. Appendix B (#appendix-b-example-strategies-for-selecting-dependencies) outlines examples policies a producer can adopt and the entities each one tracks in the refresh state.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4099684567
##########
format/view-spec.md:
##########
@@ -160,7 +180,116 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view is **fresh** when the storage table represents the result of the current view query. However, consumers may still decide to consume from a stale storage table based on their own policies.
+
+A change to the materialized view's definition produces a new `view-version-id`; any storage-table snapshot recorded at a prior `view-version-id` is invalid and should not be consumed until refreshed.
+
+#### Refresh state
+
+The refresh state record captures the state of dependencies that the producer chose to track from the materialized view's dependency graph. A dependency is recorded in `source-states` as either a `table` entry (a source table or an source materialized view's storage table) and/or a `view` entry. Source materialized views can be stored as a `view` and a `table` entry.
+
+The refresh state has the following fields:
+
+| Requirement | Field name | Description |
+|-------------|------------------------------|-------------|
+| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed |
+| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the producer chose to track; may be empty |
+| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started |
+
+##### Producer: Recording Refresh State
+
+Producers may selectively choose a subset of their dependencies to record — for example, skipping non-Iceberg sources or recording an empty list. See [Appendix B](#appendix-b-what-counts-as-a-dependency) for strategies on how to store dependency state.
+
+When writing the refresh state, producers:
+
+* **Must** record `view-version-id` and `refresh-start-timestamp-ms`.
+* **Should** include all distinct source states for the inputs they chose to track if they are reachable through multiple paths in the dependency graph.
+* **May** leave `source-states` empty (e.g., when sources are non-Iceberg or freshness is determined by a mechanism outside this spec).
+
+##### Consumer: Evaluating Refresh State
+
+Consumers may use any combination of the following to assess the state of dependencies used to produce the storage table.
+
+* **Recency policy.** Accept the storage table when `refresh-start-timestamp-ms` falls within a staleness window. A recency policy bounds data age but does not establish freshness.
+* **Trust the recorded `source-states`.** Compare each entry against the current catalog state — `snapshot-id` for tables, `version-id` for views, optionally recursive verification for source materialized views recorded by their storage tables. Also confirm that the recorded `view-version-id` equals the materialized view's current `view-version-id`.
+* **Verify by parsing the view query.** Derive the dependency set from the SQL and confirm every dependency is covered by `source-states` and matches the current state. Treat any uncovered dependency as undetermined.
+
+If a consumer's assessment passes, it reads from the storage table. If not, the consumer may fail the query, evaluate the view query directly, or apply another strategy.
Review Comment:
Similarly, this is more direct about the options, and how to approach `source-states` specifically:
```suggestion
The engine consuming a materialized view may differ from the engine that defined and refreshed it. To decide whether the storage table can be read, a consuming engine can:
* **Trust the producer's recorded state.** Evaluate freshness directly from the refresh state: compare each entry in `source-states` against the current catalog state (`snapshot-id` for tables, `version-id` for views), and confirm the recorded `view-version-id` equals the materialized view's current `view-version-id`.
* **Verify against its own derived source states.** Parse the view query, derive the dependencies the consuming engine considers relevant, and confirm the recorded `source-states` cover them and match the current state. This lets a consumer whose definition of the relevant dependencies differs from the producer's determine whether the recorded state is sufficient. Treat any uncovered dependency as undetermined.
A consumer may also apply a recency policy, accepting the storage table when `refresh-start-timestamp-ms` is within a staleness window.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1877723130
##########
format/view-spec.md:
##########
@@ -160,6 +179,56 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Full identifier
+
+The full identifier holds a reference, containing a namespace and a name, of a table or view in the catalog.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the views' catalog |
Review Comment:
Thank you!
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view that precompute the data from the view query.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`'s of the last refresh operation match the current `snapshot_id`'s of the source tables.
Review Comment:
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "wmoustafa (via GitHub)" <gi...@apache.org>.
wmoustafa commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2381029160
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the Iceberg view metadata, adding a pointer to the precomputed data and refresh information to determine if the data is still fresh.
+The refresh information is composed of data about the so-called "source tables", which are the tables referenced in the query definition of the materialized view.
+The storage table can be in the states of "fresh", "stale" or "invalid", which are determined from the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of the source tables.
+* **stale** -- The `snapshot_id`s do not match, indicating that a refresh operation needs to be performed to capture the latest source table changes.
Review Comment:
@stevenzwu, yes I prefer option 1.
@bennychow, thanks for the clarification. I agree that storing lineage information in the view is a nice-to-have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2079077777
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
Review Comment:
I agree with you that it's not a good idea to store the storage table in another catalog than the materialized view. My initial proposal didn't include the catalog. This would also get us around the issues with the "catalog alias" problem.
But I think some comments wanted to include the catalog. (I can't find it though)
I think having a dedicated namespace for all storage tables might be a use case for having a different namespace than the materialized view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2078825706
##########
format/view-spec.md:
##########
@@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _optional_ | `catalog` | A string specifying the name of the catalog. If set to `null`, the catalog is the same as the view's catalog |
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table/view |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialzied views.
+The property "refresh-state" is set on the table [snapshot summary](https://iceberg.apache.org/spec/#snapshots) to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Refresh state
+
+The refresh state record captures the state of all source tables and source views in the fully expanded query tree of the materialized view, including indirect references. Indirect references are the tables/views that are not directly referenced in the query but are nested within other views. The refresh state has the following fields:
Review Comment:
When an engine builds or refreshes the materialization, they should have the fully expanded query tree of the MV. Its easy for an engine to just write out the table snapshot ids and view version ids at that time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2317507422
CC @rdblue @danielcweeks @RussellSpitzer @szehon-ho @wmoustafa @jackye1995 @emkornfield @findepi @stevenzwu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "bennychow (via GitHub)" <gi...@apache.org>.
bennychow commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r1742296539
##########
format/view-spec.md:
##########
@@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Views have the option to be turned into materialized views by precomputing the data from the view query.
+When queried, materialized views return the precomputed data, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, known as the storage table, which stores the precomputed data.
+The metadata for a materialized view extends the common view metadata, adding a pointer to the precomputed data and freshness information to determine if the data is still fresh.
+The storage table can have the states "fresh", "stale" or "invalid".
Review Comment:
oic now, then I don't think we need to drop the sentence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "sfc-gh-ibelianski (via GitHub)" <gi...@apache.org>.
sfc-gh-ibelianski commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2593960362
##########
format/view-spec.md:
##########
@@ -63,11 +79,13 @@ The view version metadata file has the following fields:
| _required_ | `versions` | A list of known [versions](#versions) of the view [1] |
| _required_ | `version-log` | A list of [version log](#version-log) entries with the timestamp and `version-id` for every change to `current-version-id` |
| _optional_ | `properties` | A string to string map of view properties [2] |
+| _optional_ | `max-staleness-ms` | The maximum time interval in milliseconds during which changed source table snapshots are considered fresh enough to skip refreshing [3] |
Notes:
1. The number of versions to retain is controlled by the view property: `version.history.num-entries`.
2. Properties are used for metadata such as `comment` and for settings that affect view maintenance. This is not intended to be used for arbitrary metadata.
+3. The `max-staleness-ms` field only applies to materialized views and must be set to `null` for common views. When `max-staleness-ms` is not `null`, the query engine may return data directly from the `storage-table` without refreshing if all source table snapshots that differ from those used in the last refresh have timestamps within `max-staleness-ms` of the current time. When `max-staleness-ms` is `null` for a materialized view, the data in the `storage-table` is always considered fresh.
Review Comment:
this staleness definition is different from the one in the line 54 (in the current form it seem to require to NOT implement optimization that is mandated above in the line 54)
proposal:
The `max-staleness-ms`: defines staleness window for determining freshness state see above section ### Materialized Views
PS
the same problem on line 82 (contradictory definition as opposed to reference)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2631453908
##########
format/view-spec.md:
##########
@@ -160,6 +178,108 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views.
+The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table.
+
+| Requirement | Field name | Description |
+|-------------|-----------------|-------------|
+| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string |
+
+#### Freshness
+
+A materialized view's precomputed data becomes stale as the tables and views referenced in its query definition change over time. Freshness determines whether the precomputed data accurately represents the logical query definition at the current state of its dependencies.
+
+Different systems define freshness differently, based on how much of the dependency graph must be current. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, "fresh" can mean strict end-to-end consistency, acceptable lag, or policy/version compliance.
+
+A materialized view is considered fresh when its precomputed data meets the freshness criteria defined by the consumer's evaluation policy. When these criteria are not met, the materialized view is considered stale.
+
+#### Refresh state
+
+The refresh state record captures the unique dependencies in the materialized view's dependency graph. These dependencies include source Iceberg tables, views, and nested materialized views that allow a consumer to determine the freshness of the materialized view.
+
+**Producer responsibilities:**
+- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation.
+- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables).
Review Comment:
source states list may be empty if the source state cannot be determined, or producer freshness policy doesn't require for staleness determination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2594718087
##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki
Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view query stored as a table.
+When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data.
+Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata.
+Refresh metadata contains information about the "source tables" and/or "source views", which are the tables/views referenced in the query definition of the materialized view.
+During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations:
+* **fresh** -- The `snapshot_id`s of the last refresh operation match the current `snapshot_id`s of all the source tables, OR all source table snapshots that differ from the last refresh have timestamps within a configured staleness window.
Review Comment:
Thanks you very much for your improvements! I've updated the text
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r3003387100
##########
format/view-spec.md:
##########
@@ -160,7 +178,122 @@ Each entry in `version-log` is a struct with the following fields:
| _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) |
| _required_ | `version-id` | ID that `current-version-id` was set to |
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name | Description |
+|-------------|----------------|-------------|
+| _required_ | `namespace` | A list of strings for namespace levels |
+| _required_ | `name` | A string specifying the name of the table |
+
Review Comment:
This field separates logical view (current) with materialized view. MV has this extra field that establishes the connection. Otherwise, how do we know the storage table is associated with the view?
Whether the storage table should be exposed in the REST catalog is out of the scope of the MV spec. catalog can choose to skip storage tables for list tables call for consumers. MV producer would still need to be able to load and update the storage table in the catalog.
There is this problem of number of round trips for loading view or MV in REST spec
* loading a logical view requires 2 round trips: (1) load table - failed (2) load view
* load a MV requires 3 round trips with current REST spec: (1) load table -failed (2) load view (3) load storage table
I am working on a proposal of universal load endpoint for relational objects: table, view, materialized view. One round trip should be able to load all 3 types of relational objects. I probably can share next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Materialized View Spec [iceberg]
Posted by "igorbelianski-cyber (via GitHub)" <gi...@apache.org>.
igorbelianski-cyber commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r2829023985
##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields:
| _required_ | `representations` | A list of [representations](#representations) for the view definition |
| _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog |
| _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier |
+| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table |
Review Comment:
the argument above that resonates with me :
we should think if: partitions spec, sort order , last-part-id , last-seq-number ... ( required table attributes)
should be thought of as optional/ pluggable for materialized views .( IMO if we deem them mandatory for tables they should be mandatory for MVs) which means table entry must always be present
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org