You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "snazy (via GitHub)" <gi...@apache.org> on 2025/01/15 10:19:31 UTC

[I] Polaris persistence mandates the use of transactions & serializable isolation [polaris]

snazy opened a new issue, #766:
URL: https://github.com/apache/polaris/issues/766

   ### Describe the bug
   
   Despite that `PolarisMetaStoreSession` [clearly says](https://github.com/apache/polaris/blob/88994f495a844f420187b9404511504601081a1a/polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreSession.java#L45-L46) that `it [is] really easy to back this using [...] simpler KV store.` the whole code architecture of everything around persistence in Polaris requires nothing less than (relational) transactions and strong consistency across multiple rows in multiple tables.
   
   For example `org.apache.polaris.core.persistence.PolarisMetaStoreManagerImpl#writeEntity`:
   ```java
     private void writeEntity(
         @Nonnull PolarisMetaStoreSession ms,
         @Nonnull PolarisBaseEntity entity,
         boolean writeToActive) {
       ms.writeToEntities(entity);
       ms.writeToEntitiesChangeTracking(entity);
   
       if (writeToActive) {
         ms.writeToEntitiesActive(entity);
       }
     }
   ```
   works against 2-3 different _rows_ in 2-3 different _tables_, requiring that all or none of the changes succeed - this is multiplied by the number of entities being written. On top it requires that entities being read before (existing and non-existing) did not change when the tx gets committed.
   
   The "pattern" of having these different tables leaks into a lot of places. It effectively makes it rather impossible to use anything else than a relational database w/ isolation level `SERIALIZABLE`.
   
   ### To Reproduce
   
   _No response_
   
   ### Actual Behavior
   
   _No response_
   
   ### Expected Behavior
   
   _No response_
   
   ### Additional context
   
   _No response_
   
   ### System information
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Polaris persistence mandates the use of transactions & serializable isolation [polaris]

Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on issue #766:
URL: https://github.com/apache/polaris/issues/766#issuecomment-2634733932

   Wrote up some analysis and tentative proposal for the way forward that should get us pretty far with fairly minimal changes:
   
   https://docs.google.com/document/d/1U9rprj8w8-Q0SnQvRMvoVlbX996z-89eOkVwWTQaZG0/edit?tab=t.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Polaris persistence mandates the use of transactions & serializable isolation [polaris]

Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on issue #766:
URL: https://github.com/apache/polaris/issues/766#issuecomment-2610652389

   The current code structure probably exaggerates the expected transaction semantics needlessly, and a few minor adjustments should be able to significantly lighten the requirements. In particular:
   
   1. Most of the independent one-shot reads happen to be structured as `runInReadTransaction` but aren't actually depending on transaction semantics
   2. The `writeToEntities`, `writeToEntitiesChangeTracking`, `writeToEntitiesActive` triplet is actually more for serving the role of secondary indexes than truly being separate tables. There are caveats depending on how we approach vestigial plans for UNDROP functionality, but it's worth discussing whether these make sense to push down to the `MetaStoreSession` layer and even refactoring the EclipseLink one to better portray these as serving efficient queries on a single table
   3. A few operations actually do uniquely require various flavors of transactions across multiple rows or tables; we should enumerate these separately from the majority of "core" actions. Of particular interest off the top of my head:
       a. CreateCatalog relationship to a "storage integration" object and bootstrapped CatalogRoles
       b. CreatePrincipal with "principal secrets"
       c. Iceberg-level multi-table commit that uses updateEntitiesPropertiesIfNotChanged
       d. DropTable atomically adding a TaskEntity for cleanup (other DropEntity types as well technically, but the main non-trivial case is for file cleanup when PURGE=true)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org