[GitHub] [iceberg] nastra commented on issue #2319: Caching Tables in SparkCatalog via CachingCatalog by default leads to stale data

[GitBox <gi...@apache.org>]

nastra commented on issue #2319:
URL: https://github.com/apache/iceberg/issues/2319#issuecomment-935635835


   I have to agree that the default caching behavior is unintuitive and rather surprising to users (as can be seen in https://github.com/projectnessie/nessie/issues/2165). 
   Using the `NessieCatalog` with Spark, one can have a table `X` on branch `dev` and the `CachingCatalog` will eventually cache this table. Users can refer to this table also as `X@dev` and the `NessieCatalog` properly splits this name into the table name `X` and the branch name `dev` via `TableReference`. However, before this happens, we reach `CachingCatalog#loadTable(..)`, where `X@dev` is treated as a separate table, meaning that an update to table `X` will leave an outdated `X@dev` table and users referring to that will always see stale data.
   
   Unfortunately there's not much control that we have over the cache invalidation procedure to handle the described use case, so I was rather thinking that Catalogs should have a way to control their default caching behavior. For the `NessieCatalog` I think it makes more sense to disable caching by default.
   
   @aokolnychyi @rdblue could you take a look at my proposal in https://github.com/apache/iceberg/pull/3230? 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org