[jira] [Commented] (HUDI-3300) Timeline server FSViewManager should avoid inline reading for metadata file partition

["Manoj Govindassamy (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480362#comment-17480362 ] 

Manoj Govindassamy commented on HUDI-3300:
------------------------------------------

Verified the time line server - it has reuse enabled and the readers opened are retained and the files listed are cached at the higher level there by serving faster requests. But for all other cases, when the {{reuse}} readers are false, we should not be caching the reader handles at all. Today we cache the readers and then close the readers towards the end. There is a possibility of multiple non {{reuse}} requests coming at the same time and using the same readers. Fullscan/Inline scan is a totally different problem and i am not going there. In the worst case two non {{reuse}} requests can use the same reader and its latest file slice retrieved at the time of caching the readers. If for any reasons the file slice happens to move forward because of concurrent upserts, the readers wouldn't know about this and would only read the old file slices.

 

Siva: 
to my understanding, you are mostly right in your explanation. but tell me something. when 2nd reader is coming through, if latest commit time hasn't changed, why would there be new updates to the file slice.  On the contrary, If there was updates to latest file slice (with new log appends), latest commit time would have updated and so caller should have re-initialized the file system view right. or does this re-initialize happen only incase of timeline server and at other places we don't keep refreshing the fileSystemView.I know this is very tricky. we definitely need to get a good understanding of every nitty gritty detail here.
 

> Timeline server FSViewManager should avoid inline reading for metadata file partition
> -------------------------------------------------------------------------------------
>
>                 Key: HUDI-3300
>                 URL: https://issues.apache.org/jira/browse/HUDI-3300
>             Project: Apache Hudi
>          Issue Type: Task
>            Reporter: Manoj Govindassamy
>            Assignee: Ethan Guo
>            Priority: Blocker
>             Fix For: 0.11.0
>
>
> When inline reading is enabled, that is hoodie.metadata.enable.full.scan.log.files = false, MetadataMergedLogRecordReader doesn't cache the file listings records via the ExternalSpillableMap. So, every file listing will lead to re-reading of metadata files partition log and base files. Since files partition is less in size, even when inline reading is enabled, the TimelineServer should construct the FSViewManager with inline reading disabled for metadata files partition. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)