[jira] [Created] (IMPALA-11904) Data cache should support dumping metadata for reloading

["Ye Zihao (Jira)" <ji...@apache.org>]

Ye Zihao created IMPALA-11904:
---------------------------------

             Summary: Data cache should support dumping metadata for reloading
                 Key: IMPALA-11904
                 URL: https://issues.apache.org/jira/browse/IMPALA-11904
             Project: IMPALA
          Issue Type: Improvement
          Components: Backend
    Affects Versions: Impala 4.3.0
            Reporter: Ye Zihao
            Assignee: Ye Zihao


Data cache mainly includes cache metadata and cache files. The cache files are located on the disk and is responsible for storing cached data content, while the cache metadata is located in the memory and is responsible for indexing to the cache file according to the cache key.
Currently, if the impalad process exits, the cache metadata will be lost.   After the Impalad process restarts, we cannot reuse the cache file even though it is still on the disk, because there is no corresponding cache metadata for index.
If we can support dumping the cache metadata to disk when the process exits, then the next time the process starts it can be reloaded back into memory and the previous cache files can be reused. This would be helpful in a real production environment, where cache data often exceeds TB in size (per process), and loss of cache data due to a configuration change or version upgrade can take days to recover.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)