You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gravitino.apache.org by "FANNG1 (via GitHub)" <gi...@apache.org> on 2026/08/18 10:22:38 UTC
[D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
GitHub user FANNG1 created a discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
## Background
Gravitino supports Lance today through the Lance REST server and the generic lakehouse Lance table (`provider = lakehouse-generic`, `table_format = lance`). #11295 discusses the long-term architecture. This discussion is narrower: it is about how Lance table metadata should be stored, refreshed, and kept consistent in the current implementation, and about a refactoring that follows from it.
The motivation is not only performance. The current `loadTable` / `createTable` / `alterTable` implementations are coupled in a way that makes each new feature more expensive to add, and that coupling comes directly from an unclear answer to "who owns the schema".
## Scope
In scope:
- Where Lance table schema and version live, and how they are refreshed.
- How `loadTable`, `createTable` and `alterTable` are decomposed.
Out of scope — assumed as already decided in #11295:
- The Gravitino backend is auxiliary-only; standalone Lance REST does not talk to Gravitino over HTTP.
- Whether Gravitino should support path / rest namespace backends. Nothing here depends on that outcome.
## Problems
**1. Schema freshness versus cost is a deployment-wide switch.** `loadTable` can open the Lance dataset on every call to compare versions (accurate, but an object-store round trip per request — not acceptable on the Lance REST hot path), or skip the check (cheap, but the schema may be stale). Today this is chosen once per catalog, so a single setting has to serve both the engine-facing REST path and the governance-facing Gravitino REST path, which want opposite things. The choice belongs to the caller, not to the deployment.
**2. `loadTable`, `createTable` and `alterTable` are coupled.**
- `loadTable` has a write side effect: it repairs the cached schema. As a result almost every other operation deliberately bypasses it.
- `createTable` calls into load / drop / purge to implement `EXIST_OK` and `OVERWRITE`.
- `alterTable` writes the dataset first, then smuggles the resulting version into the metadata update as a synthetic table change.
- The dataset version is written from four different places, each with its own concurrency handling.
## Principle
Two source-of-truth domains, not one:
```text
schema / columns / index / dataset version -> Lance dataset (source of truth)
location / storage options / comment /
external / tags / policy / owner -> Gravitino store (source of truth)
```
This is deliberately narrower than "the Lance dataset is the source of truth". Location, credentials, ownership and governance metadata are not in the dataset and never will be.
Under this principle, Gravitino's stored columns are an explicit **cache** of the first domain. Empty and stale become legal states with a defined convergence path, rather than bugs.
This also makes explicit what @yuqi1129 already pointed out in #11295: Lance is path-self-describing, applications routinely write to the path directly, so catalog-held schema can only ever be a projection of what the path says.
## Proposed shape
**Two load paths instead of one switch.**
```text
light load entity store only, never opens the dataset
-> location, storage options, properties
-> serves describeTable when the caller does not need schema detail
full load light load + open dataset + version check, always
-> schema, indexes
-> serves describeTable with detailed metadata, and Gravitino REST
```
The version check is not configurable: light load never does it, full load always does it. The existing catalog-level schema refresh mode goes away — freshness is decided by which interface the caller uses.
Engines that only need a location never pay for a dataset open. Callers that ask for a schema always get one that matches the dataset.
**A single schema updater.**
```text
updater(table, version) bring this table's cached metadata to at least this version
idempotent, monotonic, no-op when already >= version
```
The only writer of cached schema and version. Full load calls it after reading the dataset; schema-altering operations call it after writing the dataset. The four scattered write points collapse into one, and `loadTable` stops being a mutating operation from the caller's point of view.
**Alter writes the dataset, then notifies.**
Schema changes go to the Lance dataset first, then to the updater. Non-schema changes (comment, properties, rename) stay entirely in the entity store — second domain, never touch the dataset.
## Consequences
**Out-of-band writes become a stated contract.** They are expected, the cached copy is allowed to lag, and it converges on the next full load. Because full load always checks the version, a stale cache is never observed by a caller that asks for a schema — staleness is confined to the stored copy, and light load callers have explicitly opted out of schema detail.
**This is a step toward the commit protocol, not a detour.** The updater is exactly where the long-term Lance REST commit protocol plugs in. Today it is driven by "someone read the dataset and observed version N"; later also by "a commit through Lance REST produced version N". Same interface, same monotonic semantics.
## Compatibility
`provider = lakehouse-generic` with `table_format = lance` keeps working. The changes are internal to the Lance table operations, not to the catalog contract. No metadata migration is expected: existing tables converge to the new model on their next full load.
The externally visible change is the catalog-level schema refresh mode property, which becomes unnecessary and should be deprecated. Callers that need an accurate schema get one from full load regardless of configuration, which is strictly better than what the `VERSION_CHECK` setting provides today.
GitHub link: https://github.com/apache/gravitino/discussions/12485
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
Posted by "yuqi1129 (via GitHub)" <gi...@apache.org>.
GitHub user yuqi1129 added a comment to the discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
Thanks for writing this down. I agree with the overall direction: clarifying the responsibility, boundary, and semantics of each interface is the right thing to do.
However, I think the following points need to be clarified before we move forward:
1. **Where are the light and full load paths implemented?**
Is this split implemented in Lance REST, or in `lakehouse-generic`?
- If it is implemented in Lance REST, how does it integrate with the current engine-facing path? Which concrete endpoint or interface maps to the light load and which maps to the full load?
- If it is implemented in `lakehouse-generic`, do we need to change the current catalog interface? `loadTable` is a widely used and fundamental interface, so introducing two load semantics there would be a relatively large API change.
We also need a clear caller-selection model. How does a user know which interface to use? Can the same application choose per request based on its use case, or must it select only one path? The current design does have a problem because the behavior is configured at the catalog level, so callers cannot choose freely. The proposal improves this only if the concrete API allows the caller to make that choice explicitly.
2. **`updater(table, version)`**
This contract is more precise than the current Gravitino `alter` API and can provide CAS-like, monotonic update semantics. I do not see a major problem with the interface itself. My main concern is how it will be used in practice: whether it is public or internal, who calls it, and what the concurrency, retry, and failure semantics are. It would help to show the complete call flow for load and alter operations.
3. **Create, register, and declare**
Do you plan to support all three as Gravitino operations? Are they Lance-specific APIs, operations specific to Lance tables in `lakehouse-generic`, or new generic catalog APIs? The intended scope matters: adding them to the generic Gravitino catalog interface is a much broader commitment than keeping them within the Lance-specific implementation.
GitHub link: https://github.com/apache/gravitino/discussions/12485#discussioncomment-18077683
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
Posted by "yuqi1129 (via GitHub)" <gi...@apache.org>.
GitHub user yuqi1129 added a comment to the discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
> Lance REST takes light load if load detailed metadata is false.
Does it mean that we can customize the detail option and set it to true or false to decide whether to use light or full load?
For the rest, your answer makes sense, and I'm generally okay with it.
GitHub link: https://github.com/apache/gravitino/discussions/12485#discussioncomment-18102965
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
Posted by "FANNG1 (via GitHub)" <gi...@apache.org>.
GitHub user FANNG1 edited a discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
## Background
Gravitino supports Lance today through the Lance REST server and the generic lakehouse Lance table (`provider = lakehouse-generic`, `table_format = lance`). #11295 discusses the long-term architecture. This discussion is narrower: it is about how Lance table metadata should be stored, refreshed, and kept consistent in the current implementation, and about a refactoring that follows from it.
The motivation is not only performance. The current `loadTable` / `createTable` / `alterTable` implementations are coupled in a way that makes each new feature more expensive to add, and that coupling comes directly from an unclear answer to "who owns the schema".
## Scope
In scope:
- Where Lance table schema and version live, and how they are refreshed.
- How `loadTable`, `createTable` and `alterTable` are decomposed.
Out of scope — assumed as already decided in #11295:
- The Gravitino backend is auxiliary-only; standalone Lance REST does not talk to Gravitino over HTTP.
- Whether Gravitino should support path / rest namespace backends. Nothing here depends on that outcome.
## Problems
**1. Schema freshness versus cost is a deployment-wide switch.** `loadTable` can open the Lance dataset on every call to compare versions (accurate, but an object-store round trip per request — not acceptable on the Lance REST hot path), or skip the check (cheap, but the schema may be stale). Today this is chosen once per catalog, so a single setting has to serve both the engine-facing REST path and the governance-facing Gravitino REST path, which want opposite things. The choice belongs to the caller, not to the deployment.
**2. `loadTable`, `createTable` and `alterTable` are coupled.**
- `loadTable` has a write side effect: it repairs the cached schema. As a result almost every other operation deliberately bypasses it.
- `createTable` calls into load / drop / purge to implement `EXIST_OK` and `OVERWRITE`.
- `alterTable` writes the dataset first, then smuggles the resulting version into the metadata update as a synthetic table change.
- The dataset version is written from four different places, each with its own concurrency handling.
## Principle
Two source-of-truth domains, not one:
```text
schema / columns / index / dataset version -> Lance dataset (source of truth)
location / storage options / comment /
external / tags / policy / owner -> Gravitino store (source of truth)
```
This is deliberately narrower than "the Lance dataset is the source of truth". Location, credentials, ownership and governance metadata are not in the dataset and never will be.
Under this principle, Gravitino's stored columns are an explicit **cache** of the first domain. Empty and stale become legal states with a defined convergence path, rather than bugs.
This also makes explicit what @yuqi1129 already pointed out in #11295: Lance is path-self-describing, applications routinely write to the path directly, so catalog-held schema can only ever be a projection of what the path says.
## Proposed shape
**Two load paths instead of one switch.**
```text
light load entity store only, never opens the dataset
-> location, storage options, properties
-> serves describeTable when the caller does not need schema detail
full load light load + open dataset + version check, always
-> schema, indexes
-> serves describeTable with detailed metadata, and Gravitino REST
```
The version check is not configurable: light load never does it, full load always does it. The existing catalog-level schema refresh mode goes away — freshness is decided by which interface the caller uses.
Engines that only need a location never pay for a dataset open. Callers that ask for a schema always get one that matches the dataset.
**A single schema updater.**
```text
updater(table, version) bring this table's cached metadata to at least this version
idempotent, monotonic, no-op when already >= version
```
The only writer of cached schema and version. Full load calls it after reading the dataset; schema-altering operations call it after writing the dataset. The four scattered write points collapse into one, and `loadTable` stops being a mutating operation from the caller's point of view.
**Alter writes the dataset, then notifies.**
Schema changes go to the Lance dataset first, then to the updater. Non-schema changes (comment, properties, rename) stay entirely in the entity store — second domain, never touch the dataset.
GitHub link: https://github.com/apache/gravitino/discussions/12485
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
Posted by "FANNG1 (via GitHub)" <gi...@apache.org>.
GitHub user FANNG1 edited a discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
## Background
Gravitino supports Lance today through the Lance REST server and the generic lakehouse Lance table (`provider = lakehouse-generic`, `table_format = lance`). #11295 discusses the long-term architecture. This discussion is narrower: it is about how Lance table metadata should be stored, refreshed, and kept consistent in the current implementation, and about a refactoring that follows from it.
The motivation is not only performance. The current `loadTable` / `createTable` / `alterTable` implementations are coupled in a way that makes each new feature more expensive to add, and that coupling comes directly from an unclear answer to "who owns the schema".
## Scope
In scope:
- Where Lance table schema and version live, and how they are refreshed.
- How `loadTable`, `createTable` and `alterTable` are decomposed.
Out of scope:
- Whether Gravitino should support path / rest namespace backends. Nothing here depends on that outcome.
Working assumption, not yet settled: the Gravitino backend is auxiliary-only, so standalone Lance REST does not talk to Gravitino over HTTP. This was proposed in #11295 but not concluded there — the most recent comment on that thread asks for the opposite, a standalone Lance REST server authenticating against a secured Gravitino server for parity with Iceberg REST.
It matters here because it decides where the light load path lives. Auxiliary-only lets it be an internal interface reached through Gravitino dispatchers. If the standalone-over-HTTP mode survives, the same capability has to be exposed as a public Gravitino REST endpoint instead, which is a larger commitment. The rest of this proposal holds either way.
## Problems
**1. Schema freshness versus cost is a deployment-wide switch.** `loadTable` can open the Lance dataset on every call to compare versions (accurate, but an object-store round trip per request — not acceptable on the Lance REST hot path), or skip the check (cheap, but the schema may be stale). Today this is chosen once per catalog, so a single setting has to serve both the engine-facing REST path and the governance-facing Gravitino REST path, which want opposite things. The choice belongs to the caller, not to the deployment.
**2. `loadTable`, `createTable` and `alterTable` are coupled.**
- `loadTable` has a write side effect: it repairs the cached schema. As a result almost every other operation deliberately bypasses it.
- `createTable` calls into load / drop / purge to implement `EXIST_OK` and `OVERWRITE`.
- `alterTable` writes the dataset first, then smuggles the resulting version into the metadata update as a synthetic table change.
- The dataset version is written from four different places, each with its own concurrency handling.
- `createTable` also carries three different operations — create, register and declare — told apart by flags in the table property map rather than by the interface. The set of legal creation modes therefore depends on which flags are set, and illegal combinations are caught by scattered runtime checks.
## Principle
Two source-of-truth domains, not one:
```text
schema / columns / index / dataset version -> Lance dataset (source of truth)
location / storage options / comment /
external / tags / policy / owner -> Gravitino store (source of truth)
```
This is deliberately narrower than "the Lance dataset is the source of truth". Location, credentials, ownership and governance metadata are not in the dataset and never will be.
Under this principle, Gravitino's stored columns are an explicit **cache** of the first domain. Empty and stale become legal states with a defined convergence path, rather than bugs.
This also makes explicit what @yuqi1129 already pointed out in #11295: Lance is path-self-describing, applications routinely write to the path directly, so catalog-held schema can only ever be a projection of what the path says.
## Proposed shape
**Two load paths instead of one switch.**
```text
light load entity store only, never opens the dataset
-> location, storage options, properties
-> serves describeTable when the caller does not need schema detail
full load light load + open dataset + version check, always
-> schema, indexes
-> serves describeTable with detailed metadata, and Gravitino REST
```
The version check is not configurable: light load never does it, full load always does it. The existing catalog-level schema refresh mode goes away — freshness is decided by which interface the caller uses.
Engines that only need a location never pay for a dataset open. Callers that ask for a schema always get one that matches the dataset.
**A single schema updater.**
```text
updater(table, version) bring this table's cached metadata to at least this version
idempotent, monotonic, no-op when already >= version
```
The only writer of cached schema and version. Full load calls it after reading the dataset; schema-altering operations call it after writing the dataset. The four scattered write points collapse into one, and `loadTable` stops being a mutating operation from the caller's point of view.
**Create, register and declare become separate operations.**
```text
create write the dataset + create the entity
register create the entity for a dataset that already exists
declare create the entity for a dataset that does not exist yet
```
They share the entity creation step and differ only on the dataset side. The creation mode becomes a parameter of each operation, with its legal values expressed in that operation's signature, so illegal combinations no longer need cross-property runtime checks.
Once the stored schema is a cache, register and declare differ only in whether the dataset already exists — both create an entity whose cache is empty and gets filled by the first full load. Whether the two can be merged is worth evaluating separately.
**Alter writes the dataset, then notifies.**
Schema changes go to the Lance dataset first, then to the updater. Non-schema changes (comment, properties, rename) stay entirely in the entity store — second domain, never touch the dataset.
The updater call stays synchronous within the alter request, so `alterTable` keeps returning the updated table and existing event and audit behaviour is unchanged. Because the updater is idempotent and monotonic, a failure there does not have to fail the request: the cache simply lags until the next full load.
## Consequences
**Out-of-band writes become a stated contract.** They are expected, the cached copy is allowed to lag, and it converges on the next full load. Because full load always checks the version, a stale cache is never observed by a caller that asks for a schema — staleness is confined to the stored copy, and light load callers have explicitly opted out of schema detail.
**This is a step toward the commit protocol, not a detour.** The updater is exactly where the long-term Lance REST commit protocol plugs in. Today it is driven by "someone read the dataset and observed version N"; later also by "a commit through Lance REST produced version N". Same interface, same monotonic semantics.
## Compatibility
`provider = lakehouse-generic` with `table_format = lance` keeps working. The changes are internal to the Lance table operations, not to the catalog contract. No metadata migration is expected: existing tables converge to the new model on their next full load.
The externally visible change is the catalog-level schema refresh mode property, which becomes unnecessary and should be deprecated. Callers that need an accurate schema get one from full load regardless of configuration, which is strictly better than what the `VERSION_CHECK` setting provides today.
GitHub link: https://github.com/apache/gravitino/discussions/12485
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
Posted by "FANNG1 (via GitHub)" <gi...@apache.org>.
GitHub user FANNG1 added a comment to the discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
> Does it mean that we can customize the detail option and set it to true or false to decide whether to use light or full load?
No, this is decided by the load detailed metadata parameter in describe table request, by default, it should be false.
GitHub link: https://github.com/apache/gravitino/discussions/12485#discussioncomment-18104045
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
Posted by "FANNG1 (via GitHub)" <gi...@apache.org>.
GitHub user FANNG1 added a comment to the discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
1. light and full load are APIs in [LanceTableOperations](https://github.com/apache/gravitino/blob/main/lance/lance-common/src/main/java/org/apache/gravitino/lance/common/ops/LanceTableOperations.java), Gravitino REST takes full load, Lance REST takes light load if load detailed metadata is false.
2. updater is a internal module for load/alter operations to sync schema in entity store with underlying dataset, we could invoke update operation after alter lance dataset or finding the entity store version is less than lance dataset version. the update is invoked only when the entity cache version is less than the version in lance dataset
3. I prefer to split 3 APIs in [LanceTableOperations](https://github.com/apache/gravitino/blob/main/lance/lance-common/src/main/java/org/apache/gravitino/lance/common/ops/LanceTableOperations.java), currently they are mixed and hard to maintaince.
GitHub link: https://github.com/apache/gravitino/discussions/12485#discussioncomment-18086800
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
Posted by "FANNG1 (via GitHub)" <gi...@apache.org>.
GitHub user FANNG1 edited a discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
## Background
Gravitino supports Lance today through the Lance REST server and the generic lakehouse Lance table (`provider = lakehouse-generic`, `table_format = lance`). #11295 discusses the long-term architecture. This discussion is narrower: it is about how Lance table metadata should be stored, refreshed, and kept consistent in the current implementation, and about a refactoring that follows from it.
The motivation is not only performance. The current `loadTable` / `createTable` / `alterTable` implementations are coupled in a way that makes each new feature more expensive to add, and that coupling comes directly from an unclear answer to "who owns the schema".
## Scope
In scope:
- Where Lance table schema and version live, and how they are refreshed.
- How `loadTable`, `createTable` and `alterTable` are decomposed.
Out of scope:
- Whether Gravitino should support path / rest namespace backends. Nothing here depends on that outcome.
Working assumption, not yet settled: the Gravitino backend is auxiliary-only, so standalone Lance REST does not talk to Gravitino over HTTP. This was proposed in #11295 but not concluded there — the most recent comment on that thread asks for the opposite, a standalone Lance REST server authenticating against a secured Gravitino server for parity with Iceberg REST.
It matters here because it decides where the light load path lives. Auxiliary-only lets it be an internal interface reached through Gravitino dispatchers. If the standalone-over-HTTP mode survives, the same capability has to be exposed as a public Gravitino REST endpoint instead, which is a larger commitment. The rest of this proposal holds either way.
## Problems
**1. Schema freshness versus cost is a deployment-wide switch.** `loadTable` can open the Lance dataset on every call to compare versions (accurate, but an object-store round trip per request — not acceptable on the Lance REST hot path), or skip the check (cheap, but the schema may be stale). Today this is chosen once per catalog, so a single setting has to serve both the engine-facing REST path and the governance-facing Gravitino REST path, which want opposite things. The choice belongs to the caller, not to the deployment.
**2. `loadTable`, `createTable` and `alterTable` are coupled.**
- `loadTable` has a write side effect: it repairs the cached schema. As a result almost every other operation deliberately bypasses it.
- `alterTable` writes the dataset first, then smuggles the resulting version into the metadata update as a synthetic table change.
- The dataset version is written from four different places, each with its own concurrency handling.
- `createTable` also carries three different operations — create, register and declare — told apart by flags in the table property map rather than by the interface. The set of legal creation modes therefore depends on which flags are set, and illegal combinations are caught by scattered runtime checks.
## Principle
Two source-of-truth domains, not one:
```text
schema / columns / index / dataset version -> Lance dataset (source of truth)
location / storage options / comment /
external / tags / policy / owner -> Gravitino store (source of truth)
```
This is deliberately narrower than "the Lance dataset is the source of truth". Location, credentials, ownership and governance metadata are not in the dataset and never will be.
Under this principle, Gravitino's stored columns are an explicit **cache** of the first domain. Empty and stale become legal states with a defined convergence path, rather than bugs.
This also makes explicit what @yuqi1129 already pointed out in #11295: Lance is path-self-describing, applications routinely write to the path directly, so catalog-held schema can only ever be a projection of what the path says.
## Proposed shape
**Two load paths instead of one switch.**
```text
light load entity store only, never opens the dataset
-> location, storage options, properties
-> serves describeTable when the caller does not need schema detail
full load light load + open dataset + version check, always
-> schema, indexes
-> serves describeTable with detailed metadata, and Gravitino REST
```
The version check is not configurable: light load never does it, full load always does it. The existing catalog-level schema refresh mode goes away — freshness is decided by which interface the caller uses.
Engines that only need a location never pay for a dataset open. Callers that ask for a schema always get one that matches the dataset.
**A single schema updater.**
```text
updater(table, version) bring this table's cached metadata to at least this version
idempotent, monotonic, no-op when already >= version
```
The only writer of cached schema and version. Full load calls it after reading the dataset; schema-altering operations call it after writing the dataset. The four scattered write points collapse into one, and `loadTable` stops being a mutating operation from the caller's point of view.
**Create, register and declare become separate operations.**
```text
create write the dataset + create the entity
register create the entity for a dataset that already exists
declare create the entity for a dataset that does not exist yet
```
They share the entity creation step and differ only on the dataset side. The creation mode becomes a parameter of each operation, with its legal values expressed in that operation's signature, so illegal combinations no longer need cross-property runtime checks.
Once the stored schema is a cache, register and declare differ only in whether the dataset already exists — both create an entity whose cache is empty and gets filled by the first full load. Whether the two can be merged is worth evaluating separately.
**Alter writes the dataset, then notifies.**
Schema changes go to the Lance dataset first, then to the updater. Non-schema changes (comment, properties, rename) stay entirely in the entity store — second domain, never touch the dataset.
The updater call stays synchronous within the alter request, so `alterTable` keeps returning the updated table and existing event and audit behaviour is unchanged. Because the updater is idempotent and monotonic, a failure there does not have to fail the request: the cache simply lags until the next full load.
GitHub link: https://github.com/apache/gravitino/discussions/12485
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables [gravitino]
Posted by "yuqi1129 (via GitHub)" <gi...@apache.org>.
GitHub user yuqi1129 edited a comment on the discussion: [DISCUSS] Treat the Gravitino entity store as a schema cache for Lance tables
> Lance REST takes light load if load detailed metadata is false.
Does it mean that we can customize the detail option and set it to true or false to decide whether to use light or full load?
For the rest, your answer makes sense, and I'm generally okay with it.
GitHub link: https://github.com/apache/gravitino/discussions/12485#discussioncomment-18102965
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org