[jira] [Created] (IMPALA-9841) Table level writeIdList represent the filemetadata snapshot

["Vihang Karajgaonkar (Jira)" <ji...@apache.org>]

Vihang Karajgaonkar created IMPALA-9841:
-------------------------------------------

             Summary: Table level writeIdList represent the filemetadata snapshot
                 Key: IMPALA-9841
                 URL: https://issues.apache.org/jira/browse/IMPALA-9841
             Project: IMPALA
          Issue Type: Improvement
            Reporter: Vihang Karajgaonkar
            Assignee: Vihang Karajgaonkar


Currently, in HdfsTable https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1017 we reload the ValidWriteIdList every time. However in case of incremental refresh, it is possible that the file-metadata is not loaded at all or only loaded for certain partitions. This would mean that the cached ValidWriteIdList of the table is out-of-sync with the actual file-metadata which is cached in catalog.

This is currently not a problem since we on rely on the ValidWriteIdList during the loading of the file-metadata. However, for a consistent reads of the transactional tables, we would need to make sure that this ValidWriteIdList is guaranteed to represent the cached files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org