You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2025/07/13 06:36:29 UTC

[PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

codope opened a new pull request, #395:
URL: https://github.com/apache/hudi-rs/pull/395

   ## Description
   
   Implements LSM timeline reading capability for the hudi-rs core to support both v6 and v8 timeline formats, following Java implementation patterns with timeline layout abstraction. This is the foundation PR and we will follow up with future work on LSM timeline history reading, support for multiple layers of timeline compaction and manifest parsing for compacted timeline files.
   
   ### Core Implementation
   - **Timeline Layout Architecture**: Added `TimelineLayoutType` enum to manage different timeline storage formats.
   - **Active Timeline**: Implemented `ActiveTimeline` for older table versions with layout version 1 (existing format).
   - **LSM Timeline**: Added `LsmTimeline` foundation for newer table version 8 with layout version 2 (LSM format).
   - **Factory Pattern**: Created `TimelineLayoutFactory` for creating appropriate timeline implementations based on table configuration.
   
   ```rust
   Timeline {
       layout: TimelineLayoutType::Active(ActiveTimeline) |
               TimelineLayoutType::Lsm(LsmTimeline)
   }
   ```
   
   - **layout v1**: Uses `ActiveTimeline` (reads from `.hoodie/` directory)
   - **layout v2**: Uses `LsmTimeline` (foundation for `.hoodie/timeline/` LSM format)
   
   ### Configuration Integration
   - **Version Detection**: Automatic timeline layout selection based on `hoodie.table.version` and `hoodie.timeline.layout.version`
   - **Graceful Fallback**: Handles invalid table configurations by defaulting to active timeline layout.
   - **Backward Compatibility**: Maintains full compatibility with existing v6 table APIs.
   
   <!--- Please link any related issues and PRs as well. -->
   
   Related to #8 
   
   ## How are the changes test-covered
   
   - **250/250 core tests** passing  
   - **Integration tests** for table reading, snapshot queries, and file slices
   - **Backward compatibility** verified with existing test data
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2315143086


##########
crates/core/src/timeline/mod.rs:
##########
@@ -54,75 +61,70 @@ pub const DEFAULT_LOADING_ACTIONS: &[Action] =
     &[Action::Commit, Action::DeltaCommit, Action::ReplaceCommit];
 
 impl Timeline {
-    #[cfg(test)]
-    pub(crate) async fn new_from_completed_commits(
+    pub(crate) fn new(
         hudi_configs: Arc<HudiConfigs>,
-        storage_options: Arc<HashMap<String, String>>,
-        completed_commits: Vec<Instant>,
-    ) -> Result<Self> {
-        let storage = Storage::new(storage_options.clone(), hudi_configs.clone())?;
-        Ok(Self {
+        storage: Arc<Storage>,
+        active_loader: TimelineLoader,
+        archived_loader: Option<TimelineLoader>,
+    ) -> Self {
+        Self {
             hudi_configs,
             storage,
-            completed_commits,
-        })
+            active_loader,
+            archived_loader,
+            completed_commits: Vec::new(),
+        }
     }
 
     pub(crate) async fn new_from_storage(
         hudi_configs: Arc<HudiConfigs>,
         storage_options: Arc<HashMap<String, String>>,
     ) -> Result<Self> {
         let storage = Storage::new(storage_options.clone(), hudi_configs.clone())?;
+        let mut timeline = TimelineBuilder::new(hudi_configs, storage).build().await?;
         let selector = TimelineSelector::completed_actions_in_range(
             DEFAULT_LOADING_ACTIONS,
-            hudi_configs.clone(),
+            timeline.hudi_configs.clone(),
             None,
             None,
         )?;
-        let completed_commits = Self::load_instants(&selector, &storage, false).await?;
-        Ok(Self {
-            hudi_configs,
-            storage,
-            completed_commits,
-        })
+        timeline.completed_commits = timeline.load_instants(&selector, false).await?;
+        Ok(timeline)
+    }
+
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        self.active_loader.load_instants(selector, desc).await
     }
 
-    async fn load_instants(
+    pub async fn load_instants_with_archive(
+        &self,
         selector: &TimelineSelector,
-        storage: &Storage,
         desc: bool,
+        include_archived: bool,

Review Comment:
   Agreed. We’ll keep public API simple and make archived loading an internal concern driven by the selector range; removed the public flag for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2315147424


##########
crates/core/src/timeline/mod.rs:
##########
@@ -54,75 +61,70 @@ pub const DEFAULT_LOADING_ACTIONS: &[Action] =
     &[Action::Commit, Action::DeltaCommit, Action::ReplaceCommit];
 
 impl Timeline {
-    #[cfg(test)]
-    pub(crate) async fn new_from_completed_commits(
+    pub(crate) fn new(
         hudi_configs: Arc<HudiConfigs>,
-        storage_options: Arc<HashMap<String, String>>,
-        completed_commits: Vec<Instant>,
-    ) -> Result<Self> {
-        let storage = Storage::new(storage_options.clone(), hudi_configs.clone())?;
-        Ok(Self {
+        storage: Arc<Storage>,
+        active_loader: TimelineLoader,
+        archived_loader: Option<TimelineLoader>,
+    ) -> Self {
+        Self {
             hudi_configs,
             storage,
-            completed_commits,
-        })
+            active_loader,
+            archived_loader,
+            completed_commits: Vec::new(),
+        }
     }
 
     pub(crate) async fn new_from_storage(
         hudi_configs: Arc<HudiConfigs>,
         storage_options: Arc<HashMap<String, String>>,
     ) -> Result<Self> {
         let storage = Storage::new(storage_options.clone(), hudi_configs.clone())?;
+        let mut timeline = TimelineBuilder::new(hudi_configs, storage).build().await?;
         let selector = TimelineSelector::completed_actions_in_range(
             DEFAULT_LOADING_ACTIONS,
-            hudi_configs.clone(),
+            timeline.hudi_configs.clone(),
             None,
             None,
         )?;
-        let completed_commits = Self::load_instants(&selector, &storage, false).await?;
-        Ok(Self {
-            hudi_configs,
-            storage,
-            completed_commits,
-        })
+        timeline.completed_commits = timeline.load_instants(&selector, false).await?;
+        Ok(timeline)
+    }
+
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        self.active_loader.load_instants(selector, desc).await
     }
 
-    async fn load_instants(
+    pub async fn load_instants_with_archive(
+        &self,
         selector: &TimelineSelector,
-        storage: &Storage,
         desc: bool,
+        include_archived: bool,
     ) -> Result<Vec<Instant>> {
-        let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
-
-        // For most cases, we load completed instants, so we can pre-allocate the vector with a
-        // capacity of 1/3 of the total number of listed files,
-        // ignoring requested and inflight instants.
-        let mut instants = Vec::with_capacity(files.len() / 3);
-
-        for file_info in files {
-            match selector.try_create_instant(file_info.name.as_str()) {
-                Ok(instant) => instants.push(instant),
-                Err(e) => {
-                    // Ignore files that are not valid or desired instants.
-                    debug!(
-                        "Instant not created from file {:?} due to: {:?}",
-                        file_info, e
-                    );
+        // Always try active first
+        let mut instants = self.active_loader.load_instants(selector, desc).await?;
+
+        // Load archived/compacted if flag is set
+        if include_archived {
+            if let Some(archived_loader) = &self.archived_loader {
+                let archived_instants = archived_loader
+                    .load_archived_instants(selector, desc)
+                    .await?;
+                instants.extend(archived_instants);
+
+                // Re-sort after merging
+                instants.sort_unstable();

Review Comment:
   Loaders now return sorted lists; we'll replace the final sort with a merge when archived reading lands.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2384494738


##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::internal::HudiInternalConfig::{
+    TimelineArchivedReadEnabled, TimelineArchivedUnavailableBehavior,
+};
+use crate::config::table::HudiTableConfig::ArchiveLogFolder;
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoArchived(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {

Review Comment:
   @codope this is the only blocker for merging. if we need to pass v8 timeline loading test cases, this parsing needs to be there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2203215042


##########
crates/core/src/file_group/reader.rs:
##########
@@ -75,8 +75,8 @@ impl FileGroupReader {
     /// Creates a new reader with the given base URI and options.
     ///
     /// # Arguments
-    ///     * `base_uri` - The base URI of the file group's residing table.
-    ///     * `options` - Additional options for the reader.
+    /// * `base_uri` - The base URI of the file group's residing table.
+    /// * `options` - Additional options for the reader.

Review Comment:
   Please ignore these changes while reviewing. When I ran `cargo test --workspace` there were quite a few doctests that failed locally. So, I had to fix them.



##########
crates/core/src/timeline/lsm.rs:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::HudiConfigs;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+/// LSM timeline for table version 8.
+/// For now, falls back to reading from .hoodie/ directory like active timeline.
+/// TODO: Add LSM timeline history reading from .hoodie/timeline/ directory
+#[derive(Debug, Clone)]
+pub struct LsmTimeline {
+    storage: Arc<Storage>,
+}
+
+impl LsmTimeline {
+    pub fn new(_hudi_configs: Arc<HudiConfigs>, storage: Arc<Storage>) -> Self {
+        Self { storage }
+    }
+
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        // For now, fall back to reading from .hoodie/ directory like active timeline

Review Comment:
   Please ignore codecov warnings. We will implement more functionality once #394 lands. For now, we just fallback to v6.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2296780637


##########
crates/core/src/timeline/mod.rs:
##########
@@ -54,75 +61,70 @@ pub const DEFAULT_LOADING_ACTIONS: &[Action] =
     &[Action::Commit, Action::DeltaCommit, Action::ReplaceCommit];
 
 impl Timeline {
-    #[cfg(test)]
-    pub(crate) async fn new_from_completed_commits(
+    pub(crate) fn new(
         hudi_configs: Arc<HudiConfigs>,
-        storage_options: Arc<HashMap<String, String>>,
-        completed_commits: Vec<Instant>,
-    ) -> Result<Self> {
-        let storage = Storage::new(storage_options.clone(), hudi_configs.clone())?;
-        Ok(Self {
+        storage: Arc<Storage>,
+        active_loader: TimelineLoader,
+        archived_loader: Option<TimelineLoader>,
+    ) -> Self {
+        Self {
             hudi_configs,
             storage,
-            completed_commits,
-        })
+            active_loader,
+            archived_loader,
+            completed_commits: Vec::new(),
+        }
     }
 
     pub(crate) async fn new_from_storage(
         hudi_configs: Arc<HudiConfigs>,
         storage_options: Arc<HashMap<String, String>>,
     ) -> Result<Self> {
         let storage = Storage::new(storage_options.clone(), hudi_configs.clone())?;
+        let mut timeline = TimelineBuilder::new(hudi_configs, storage).build().await?;
         let selector = TimelineSelector::completed_actions_in_range(
             DEFAULT_LOADING_ACTIONS,
-            hudi_configs.clone(),
+            timeline.hudi_configs.clone(),
             None,
             None,
         )?;
-        let completed_commits = Self::load_instants(&selector, &storage, false).await?;
-        Ok(Self {
-            hudi_configs,
-            storage,
-            completed_commits,
-        })
+        timeline.completed_commits = timeline.load_instants(&selector, false).await?;
+        Ok(timeline)
+    }
+
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        self.active_loader.load_instants(selector, desc).await
     }
 
-    async fn load_instants(
+    pub async fn load_instants_with_archive(
+        &self,
         selector: &TimelineSelector,
-        storage: &Storage,
         desc: bool,
+        include_archived: bool,

Review Comment:
   I understand the current TimelineSelector has this property too, which was my initial thought. But let's try to go with simple UX for the API: callers do not need to know if they need to load archived or not, which is timeline's internal impl - they just ask for a list of instants for a given time range (everything else that a timeline selector holds):
   - if no specific time given, load active instants
   - if a start or end time given, and falls into archived range, then load relevant archived + active instants
   
   Timeline loader can internally decide what to load.
   
   The only case that we might need users to set `include_archived` is to set it `false` when they might load for a time range that could be in archived but for perf reason they choose to skip archived. This is a rare advanced usage we should be good to ignore now. 
   



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoCompacted(Arc<Storage>),

Review Comment:
   ```suggestion
       LayoutTwoArchived(Arc<Storage>),
   ```
   
   i wasn't thinking thoroughly. i think we should stick with `LayoutTwoArchived` like for layout 1, since it's still archived timeline, just in the form of LSM tree and compacted, which is internal impl.



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoCompacted(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.starts_with('_') {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+                Ok(instants)
+            }
+            _ => Err(CoreError::Unsupported(
+                "Loading from this timeline layout is not implemented yet.".to_string(),
+            )),
+        }
+    }
+
+    pub async fn load_archived_instants(
+        &self,
+        _selector: &TimelineSelector,
+        _desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneArchived(_) => Err(CoreError::Unsupported(
+                "Archived timeline for layout v1 not implemented yet".to_string(),
+            )),
+            TimelineLoader::LayoutTwoCompacted(_) => Err(CoreError::Unsupported(
+                "Compacted timeline for layout v2 not implemented yet".to_string(),

Review Comment:
   totally ok to do archived timeline support separately. reminder to fix the name as it's still archived in V2. also this method may not be pub for hiding the impl.



##########
crates/core/src/timeline/lsm_tree.rs:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::storage::Storage;
+use crate::Result;
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+
+pub const LSM_TIMELINE_DIR: &str = ".hoodie/timeline";

Review Comment:
   we need to respect `"hoodie.timeline.history.path"`, and `"hoodie.timeline.history.path"` right? same for v6, we need to handle `"hoodie.archivelog.folder"`. but we can deal with these in PR for archived timeline. the corresponding timeline loader should be responsible to take these from configs and own it



##########
crates/core/src/timeline/lsm_tree.rs:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::storage::Storage;
+use crate::Result;
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+
+pub const LSM_TIMELINE_DIR: &str = ".hoodie/timeline";
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct TimelineManifest {
+    pub version: i64,
+    pub entries: Vec<ManifestEntry>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ManifestEntry {
+    pub file_name: String,
+    pub min_instant: String,
+    pub max_instant: String,
+    pub level: i32,
+    pub file_size: i64,
+}
+
+pub struct LSMTree {
+    storage: Arc<Storage>,

Review Comment:
   need to take any hudi configs for LSM tree specific settings?



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoCompacted(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.starts_with('_') {

Review Comment:
   if we can enhance the `storage.list_files()` to support a listing for only immediate files under the prefix, then we dont need explicit check and just grab the instant files. What is the case with having `_` prefix?



##########
crates/core/src/timeline/builder.rs:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::table::HudiTableConfig::{TableVersion, TimelineLayoutVersion};
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::storage::Storage;
+use crate::timeline::loader::TimelineLoader;
+use crate::timeline::Timeline;
+use crate::Result;
+use std::sync::Arc;
+
+pub struct TimelineBuilder {
+    hudi_configs: Arc<HudiConfigs>,
+    storage: Arc<Storage>,
+}
+
+impl TimelineBuilder {
+    pub fn new(hudi_configs: Arc<HudiConfigs>, storage: Arc<Storage>) -> Self {
+        Self {
+            hudi_configs,
+            storage,
+        }
+    }
+
+    pub async fn build(self) -> Result<Timeline> {
+        let (active_loader, archived_loader) = self.resolve_loader_config()?;
+        let timeline = Timeline::new(
+            self.hudi_configs,
+            self.storage,
+            active_loader,
+            archived_loader,
+        );
+        Ok(timeline)
+    }
+
+    fn resolve_loader_config(&self) -> Result<(TimelineLoader, Option<TimelineLoader>)> {
+        let table_version = match self.hudi_configs.get(TableVersion) {
+            Ok(v) => v.to::<isize>(),
+            Err(_) => {
+                return Ok((
+                    TimelineLoader::LayoutOneActive(self.storage.clone()),
+                    Some(TimelineLoader::LayoutOneArchived(self.storage.clone())),
+                ))
+            }
+        };
+
+        let layout_version = self
+            .hudi_configs
+            .try_get(TimelineLayoutVersion)
+            .map(|v| v.to::<isize>())
+            .unwrap_or_else(|| if table_version == 8 { 2 } else { 1 });
+
+        match layout_version {
+            1 => {
+                if !(5..=8).contains(&table_version) {

Review Comment:
   reminder- master version dropped v5 support



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoCompacted(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.starts_with('_') {

Review Comment:
   it also occurs to me that we may need to enhance the listing by ignoring `.crc` files by default.. we'll track this separately.



##########
crates/core/src/timeline/lsm_tree.rs:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::storage::Storage;
+use crate::Result;
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+
+pub const LSM_TIMELINE_DIR: &str = ".hoodie/timeline";
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct TimelineManifest {
+    pub version: i64,
+    pub entries: Vec<ManifestEntry>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ManifestEntry {

Review Comment:
   maybe some docs help explain this struct



##########
crates/core/src/timeline/mod.rs:
##########
@@ -54,75 +61,70 @@ pub const DEFAULT_LOADING_ACTIONS: &[Action] =
     &[Action::Commit, Action::DeltaCommit, Action::ReplaceCommit];
 
 impl Timeline {
-    #[cfg(test)]
-    pub(crate) async fn new_from_completed_commits(
+    pub(crate) fn new(
         hudi_configs: Arc<HudiConfigs>,
-        storage_options: Arc<HashMap<String, String>>,
-        completed_commits: Vec<Instant>,
-    ) -> Result<Self> {
-        let storage = Storage::new(storage_options.clone(), hudi_configs.clone())?;
-        Ok(Self {
+        storage: Arc<Storage>,
+        active_loader: TimelineLoader,
+        archived_loader: Option<TimelineLoader>,
+    ) -> Self {
+        Self {
             hudi_configs,
             storage,
-            completed_commits,
-        })
+            active_loader,
+            archived_loader,
+            completed_commits: Vec::new(),
+        }
     }
 
     pub(crate) async fn new_from_storage(
         hudi_configs: Arc<HudiConfigs>,
         storage_options: Arc<HashMap<String, String>>,
     ) -> Result<Self> {
         let storage = Storage::new(storage_options.clone(), hudi_configs.clone())?;
+        let mut timeline = TimelineBuilder::new(hudi_configs, storage).build().await?;
         let selector = TimelineSelector::completed_actions_in_range(
             DEFAULT_LOADING_ACTIONS,
-            hudi_configs.clone(),
+            timeline.hudi_configs.clone(),
             None,
             None,
         )?;
-        let completed_commits = Self::load_instants(&selector, &storage, false).await?;
-        Ok(Self {
-            hudi_configs,
-            storage,
-            completed_commits,
-        })
+        timeline.completed_commits = timeline.load_instants(&selector, false).await?;
+        Ok(timeline)
+    }
+
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        self.active_loader.load_instants(selector, desc).await
     }
 
-    async fn load_instants(
+    pub async fn load_instants_with_archive(
+        &self,
         selector: &TimelineSelector,
-        storage: &Storage,
         desc: bool,
+        include_archived: bool,
     ) -> Result<Vec<Instant>> {
-        let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
-
-        // For most cases, we load completed instants, so we can pre-allocate the vector with a
-        // capacity of 1/3 of the total number of listed files,
-        // ignoring requested and inflight instants.
-        let mut instants = Vec::with_capacity(files.len() / 3);
-
-        for file_info in files {
-            match selector.try_create_instant(file_info.name.as_str()) {
-                Ok(instant) => instants.push(instant),
-                Err(e) => {
-                    // Ignore files that are not valid or desired instants.
-                    debug!(
-                        "Instant not created from file {:?} due to: {:?}",
-                        file_info, e
-                    );
+        // Always try active first
+        let mut instants = self.active_loader.load_instants(selector, desc).await?;
+
+        // Load archived/compacted if flag is set
+        if include_archived {
+            if let Some(archived_loader) = &self.archived_loader {
+                let archived_instants = archived_loader
+                    .load_archived_instants(selector, desc)
+                    .await?;
+                instants.extend(archived_instants);
+
+                // Re-sort after merging
+                instants.sort_unstable();

Review Comment:
   can we be sure about loading results' order- archived instants already sorted right? then we can just concat with active ones?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2315149238


##########
crates/core/src/timeline/builder.rs:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::table::HudiTableConfig::{TableVersion, TimelineLayoutVersion};
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::storage::Storage;
+use crate::timeline::loader::TimelineLoader;
+use crate::timeline::Timeline;
+use crate::Result;
+use std::sync::Arc;
+
+pub struct TimelineBuilder {
+    hudi_configs: Arc<HudiConfigs>,
+    storage: Arc<Storage>,
+}
+
+impl TimelineBuilder {
+    pub fn new(hudi_configs: Arc<HudiConfigs>, storage: Arc<Storage>) -> Self {
+        Self {
+            hudi_configs,
+            storage,
+        }
+    }
+
+    pub async fn build(self) -> Result<Timeline> {
+        let (active_loader, archived_loader) = self.resolve_loader_config()?;
+        let timeline = Timeline::new(
+            self.hudi_configs,
+            self.storage,
+            active_loader,
+            archived_loader,
+        );
+        Ok(timeline)
+    }
+
+    fn resolve_loader_config(&self) -> Result<(TimelineLoader, Option<TimelineLoader>)> {
+        let table_version = match self.hudi_configs.get(TableVersion) {
+            Ok(v) => v.to::<isize>(),
+            Err(_) => {
+                return Ok((
+                    TimelineLoader::LayoutOneActive(self.storage.clone()),
+                    Some(TimelineLoader::LayoutOneArchived(self.storage.clone())),
+                ))
+            }
+        };
+
+        let layout_version = self
+            .hudi_configs
+            .try_get(TimelineLayoutVersion)
+            .map(|v| v.to::<isize>())
+            .unwrap_or_else(|| if table_version == 8 { 2 } else { 1 });
+
+        match layout_version {
+            1 => {
+                if !(5..=8).contains(&table_version) {

Review Comment:
   updated builder to drop v5 support



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2315146326


##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoCompacted(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.starts_with('_') {

Review Comment:
   Ensured sorted results; removed `_` filter; ignoring `.crc`. Will do a shallow-listing enhancement on Storage in a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2384475925


##########
crates/core/src/timeline/selector.rs:
##########
@@ -523,24 +533,33 @@ mod tests {
     }
 
     async fn create_test_timeline() -> Timeline {
-        let instants = vec![
+        let storage = Storage::new(
+            Arc::new(HashMap::new()),
+            Arc::new(HudiConfigs::new([
+                (HudiTableConfig::BasePath, "file:///tmp/base".to_string()),
+                (HudiTableConfig::TableVersion, "6".to_string()),
+            ])),
+        )
+        .unwrap();
+        let mut timeline = TimelineBuilder::new(
+            Arc::new(HudiConfigs::new([
+                (HudiTableConfig::BasePath, "file:///tmp/base".to_string()),
+                (HudiTableConfig::TableVersion, "6".to_string()),

Review Comment:
   ```suggestion
                   (HudiTableConfig::BasePath, "file:///tmp/base"),
                   (HudiTableConfig::TableVersion, "6"),
   ```
   
   same as above. the api supports `Into<String>()`



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::internal::HudiInternalConfig::{
+    TimelineArchivedReadEnabled, TimelineArchivedUnavailableBehavior,
+};
+use crate::config::table::HudiTableConfig::ArchiveLogFolder;
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoArchived(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {

Review Comment:
   i think this API needs to handle v8 instants as the completed ones have completion ts



##########
crates/core/src/config/table.rs:
##########
@@ -166,6 +174,9 @@ impl ConfigParser for HudiTableConfig {
             Self::TimelineTimezone => Some(HudiConfigValue::String(
                 TimelineTimezoneValue::UTC.as_ref().to_string(),
             )),
+            // Fallbacks are handled in callers; no defaults here
+            Self::ArchiveLogFolder => None,
+            Self::TimelineHistoryPath => None,

Review Comment:
   these 2 do have default values right? why not provide them through this API? 



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::internal::HudiInternalConfig::{
+    TimelineArchivedReadEnabled, TimelineArchivedUnavailableBehavior,
+};
+use crate::config::table::HudiTableConfig::ArchiveLogFolder;
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoArchived(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {

Review Comment:
   ```suggestion
                       // TODO: make `storage.list_files` api support such filtering, like ignore crc and return files only
                       if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {
   ```



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::internal::HudiInternalConfig::{
+    TimelineArchivedReadEnabled, TimelineArchivedUnavailableBehavior,
+};
+use crate::config::table::HudiTableConfig::ArchiveLogFolder;
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoArchived(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            _ => Err(CoreError::Unsupported(
+                "Loading from this timeline layout is not implemented yet.".to_string(),
+            )),
+        }
+    }
+
+    pub(crate) async fn load_archived_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        // Config wiring: if archived read not enabled, return behavior based on policy
+        let storage = match self {
+            TimelineLoader::LayoutOneArchived(storage)
+            | TimelineLoader::LayoutTwoArchived(storage) => storage.clone(),
+            _ => return Ok(Vec::new()), // Active loaders don't have archived parts
+        };
+
+        let configs: Arc<HudiConfigs> = storage.hudi_configs.clone();
+        let enabled = configs
+            .get_or_default(TimelineArchivedReadEnabled)
+            .to::<bool>();
+        if !enabled {
+            let behavior = configs
+                .get_or_default(TimelineArchivedUnavailableBehavior)
+                .to::<String>()
+                .to_ascii_lowercase();
+            return match behavior.as_str() {
+                "error" => Err(CoreError::Unsupported(
+                    "Archived timeline read is disabled; shorten time range or enable archived read"
+                        .to_string(),
+                )),
+                _ => Ok(Vec::new()), // continue silently with empty archived
+            };
+        }
+
+        match self {
+            TimelineLoader::LayoutOneArchived(storage) => {
+                // Resolve archive folder from configs or fallback
+                let archive_dir = configs
+                    .try_get(ArchiveLogFolder)
+                    .map(|v| v.to::<String>())
+                    .unwrap_or_else(|| ".hoodie/archived".to_string());
+
+                // List files and try creating instants through selector
+                let files = storage.list_files(Some(&archive_dir)).await?;
+                let mut instants = Vec::new();
+                for file_info in files {
+                    if file_info.name.ends_with(".crc") {
+                        continue;
+                    }

Review Comment:
   i think we can skip checking `.crc` in this pr for simplicity. there is an open PR addresses this from storage level 



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::internal::HudiInternalConfig::{
+    TimelineArchivedReadEnabled, TimelineArchivedUnavailableBehavior,
+};
+use crate::config::table::HudiTableConfig::ArchiveLogFolder;
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoArchived(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            _ => Err(CoreError::Unsupported(
+                "Loading from this timeline layout is not implemented yet.".to_string(),
+            )),
+        }
+    }
+
+    pub(crate) async fn load_archived_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        // Config wiring: if archived read not enabled, return behavior based on policy
+        let storage = match self {
+            TimelineLoader::LayoutOneArchived(storage)
+            | TimelineLoader::LayoutTwoArchived(storage) => storage.clone(),
+            _ => return Ok(Vec::new()), // Active loaders don't have archived parts
+        };

Review Comment:
   this storage clone does not need happen here?



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::internal::HudiInternalConfig::{
+    TimelineArchivedReadEnabled, TimelineArchivedUnavailableBehavior,
+};
+use crate::config::table::HudiTableConfig::ArchiveLogFolder;
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoArchived(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            _ => Err(CoreError::Unsupported(
+                "Loading from this timeline layout is not implemented yet.".to_string(),
+            )),
+        }
+    }
+
+    pub(crate) async fn load_archived_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        // Config wiring: if archived read not enabled, return behavior based on policy
+        let storage = match self {
+            TimelineLoader::LayoutOneArchived(storage)
+            | TimelineLoader::LayoutTwoArchived(storage) => storage.clone(),
+            _ => return Ok(Vec::new()), // Active loaders don't have archived parts
+        };
+
+        let configs: Arc<HudiConfigs> = storage.hudi_configs.clone();
+        let enabled = configs
+            .get_or_default(TimelineArchivedReadEnabled)
+            .to::<bool>();
+        if !enabled {
+            let behavior = configs
+                .get_or_default(TimelineArchivedUnavailableBehavior)
+                .to::<String>()
+                .to_ascii_lowercase();
+            return match behavior.as_str() {
+                "error" => Err(CoreError::Unsupported(
+                    "Archived timeline read is disabled; shorten time range or enable archived read"
+                        .to_string(),
+                )),
+                _ => Ok(Vec::new()), // continue silently with empty archived
+            };
+        }
+
+        match self {
+            TimelineLoader::LayoutOneArchived(storage) => {
+                // Resolve archive folder from configs or fallback
+                let archive_dir = configs
+                    .try_get(ArchiveLogFolder)
+                    .map(|v| v.to::<String>())
+                    .unwrap_or_else(|| ".hoodie/archived".to_string());

Review Comment:
   here this default value can be provided through `default_value()` ? the `try_get()` API handles it



##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::internal::HudiInternalConfig::{
+    TimelineArchivedReadEnabled, TimelineArchivedUnavailableBehavior,
+};
+use crate::config::table::HudiTableConfig::ArchiveLogFolder;
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoArchived(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            _ => Err(CoreError::Unsupported(
+                "Loading from this timeline layout is not implemented yet.".to_string(),
+            )),
+        }
+    }
+
+    pub(crate) async fn load_archived_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        // Config wiring: if archived read not enabled, return behavior based on policy
+        let storage = match self {
+            TimelineLoader::LayoutOneArchived(storage)
+            | TimelineLoader::LayoutTwoArchived(storage) => storage.clone(),
+            _ => return Ok(Vec::new()), // Active loaders don't have archived parts
+        };
+
+        let configs: Arc<HudiConfigs> = storage.hudi_configs.clone();
+        let enabled = configs
+            .get_or_default(TimelineArchivedReadEnabled)
+            .to::<bool>();
+        if !enabled {
+            let behavior = configs
+                .get_or_default(TimelineArchivedUnavailableBehavior)

Review Comment:
   looks like we just need 1 config? `TimelineArchivedUnavailableBehavior` default to continue will just either load archived or do nothing. even for internal config, we need to keep it simple for easy maintenance



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2315146982


##########
crates/core/src/timeline/lsm_tree.rs:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::storage::Storage;
+use crate::Result;
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+
+pub const LSM_TIMELINE_DIR: &str = ".hoodie/timeline";
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct TimelineManifest {
+    pub version: i64,
+    pub entries: Vec<ManifestEntry>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ManifestEntry {
+    pub file_name: String,
+    pub min_instant: String,
+    pub max_instant: String,
+    pub level: i32,
+    pub file_size: i64,
+}
+
+pub struct LSMTree {
+    storage: Arc<Storage>,

Review Comment:
   We'll pass configs into the LSM reader when adding archived reads to honor timeline-specific settings.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2204765104


##########
crates/core/src/timeline/layout.rs:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::HudiConfigs;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use std::sync::Arc;
+
+/// Timeline layout implementations for different table versions.
+#[derive(Debug, Clone)]
+pub enum TimelineLayoutType {
+    Active(super::active::ActiveTimeline),
+    Lsm(super::lsm::LsmTimeline),

Review Comment:
   @codope cool! Looking closer - so `load_instants()` is actually a more generic helper function, it's the using of a `selector` in one of the init functions (`new_from_storage()`) that defines the behavior of timeline's instants loading. So depends on where `load_instants()` gets called, we just need to pass in the desired `TimelineSelector`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2452316480


##########
crates/core/src/config/table.rs:
##########
@@ -166,6 +174,9 @@ impl ConfigParser for HudiTableConfig {
             Self::TimelineTimezone => Some(HudiConfigValue::String(
                 TimelineTimezoneValue::UTC.as_ref().to_string(),
             )),
+            // Fallbacks are handled in callers; no defaults here
+            Self::ArchiveLogFolder => None,
+            Self::TimelineHistoryPath => None,

Review Comment:
   yes, updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2315148213


##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoCompacted(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.starts_with('_') {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+                Ok(instants)
+            }
+            _ => Err(CoreError::Unsupported(
+                "Loading from this timeline layout is not implemented yet.".to_string(),
+            )),
+        }
+    }
+
+    pub async fn load_archived_instants(
+        &self,
+        _selector: &TimelineSelector,
+        _desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneArchived(_) => Err(CoreError::Unsupported(
+                "Archived timeline for layout v1 not implemented yet".to_string(),
+            )),
+            TimelineLoader::LayoutTwoCompacted(_) => Err(CoreError::Unsupported(
+                "Compacted timeline for layout v2 not implemented yet".to_string(),

Review Comment:
   Made archived loading methods non-public and aligned messages to "archived" for v2 as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2204765104


##########
crates/core/src/timeline/layout.rs:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::HudiConfigs;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use std::sync::Arc;
+
+/// Timeline layout implementations for different table versions.
+#[derive(Debug, Clone)]
+pub enum TimelineLayoutType {
+    Active(super::active::ActiveTimeline),
+    Lsm(super::lsm::LsmTimeline),

Review Comment:
   @codope cool! keeping lsm tree specific in `lsm_tree.rs` is good for now. may not need to worry about re-using it for now - it can be used by the loaders exclusively, but somewhat positioned for future re-used
   
   Looking closer - so `load_instants()` is actually a more generic helper function, it's the using of a `selector` in one of the init functions (`new_from_storage()`) that defines the behavior of timeline's instants loading. So depends on where `load_instants()` gets called, we just need to pass in the desired `TimelineSelector`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#issuecomment-3341160979

   resuming review on this over the next 2 days


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add table v8+ timeline support for loading old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #395:
URL: https://github.com/apache/hudi-rs/pull/395


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2203937373


##########
crates/core/src/timeline/layout.rs:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::HudiConfigs;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use std::sync::Arc;
+
+/// Timeline layout implementations for different table versions.
+#[derive(Debug, Clone)]
+pub enum TimelineLayoutType {
+    Active(super::active::ActiveTimeline),
+    Lsm(super::lsm::LsmTimeline),

Review Comment:
   We think of `Timeline` as a top level interface where callers can interact with, the layout is an internal config that resolved based on table version and timeline layout version (as you resolved in `create_layout()`. It'd be a bit odd to see a "Type" struct having a Timeline instance basically for choosing a Timeline to use.
   
   How about we design it like:
   
   - `Timeline` remains as the top-level interface
   - When a Timeline is created, it resolves `HudiConfigs`, and create an internal model call `TimelineLoader` based on the configs
   - `TimelineLoader` can be an enum, it has these items:
   
   ```
   LayoutOneActive
   LayoutOneArchived
   LayoutTwoActive
   LayoutTwoCompacted
   ```
   
   Each loader will implement the logic to load its intended commit files.
   
   - When some APIs are invoked and need to load archived or compacted commits, the Timeline instance will use the needed `Loader`s to do IO
   - `Timeline` is responsible for collecting and stitching the `Loader`s' return data and returning final data to callers
   - We also avoid calling a LSM timeline since LSM tree is just one impl of Timeline which should be hidden away from Timeline's level



##########
crates/core/src/timeline/lsm.rs:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::HudiConfigs;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+/// LSM timeline for table version 8.
+/// For now, falls back to reading from .hoodie/ directory like active timeline.
+/// TODO: Add LSM timeline history reading from .hoodie/timeline/ directory
+#[derive(Debug, Clone)]
+pub struct LsmTimeline {
+    storage: Arc<Storage>,
+}
+
+impl LsmTimeline {

Review Comment:
   We can keep LSM tree related logic in here. As commented above, LSM tree is internal to `TimelineLoader::LayoutTwoActive, LayoutTwoCompacted`, so just those 2 loaders can make use of the logic here. Btw, the correct name is LSM tree, so let's keep it in full like `lsm_tree.rs` and `LSMTree` for other applicable names ?



##########
cpp/Cargo.toml:
##########
@@ -28,7 +28,7 @@ homepage.workspace = true
 repository = "https://github.com/apache/hudi-rs/tree/main/cpp/"
 
 [lib]
-name = "hudi"
+name = "hudi_cpp"

Review Comment:
   this actually impact the library name which is used in CMakeLists.txt . let's keep unrelated change out from the PR anyway. We try to keep artifact name minimum: we don't need to call it hudi_cpp as a c++ lib, just like we don't need to call the python artifact pyhudi - we just `pip install hudi`



##########
crates/core/src/timeline/active.rs:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::HudiConfigs;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+/// Active timeline for table versions 5-8.
+/// Reads individual instant files from `.hoodie/` directory.
+#[derive(Debug, Clone)]
+pub struct ActiveTimeline {
+    storage: Arc<Storage>,
+}
+
+impl ActiveTimeline {

Review Comment:
   let's take the opportunity to re-think some of the model design: 
   
   from the callers perspective,  they should not care if the commits should come from active timeline `history/` or `archived/` because these are all timeline internal logic flow. If the caller requests require looking into compacted or archived commits, the timeline instance will need to make a call and fetch the data as needed. So at the interface level, we don't need an explicit `ActiveTimeline`. We can just keep it `Timeline` as the front-facing model.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2315149878


##########
crates/core/src/timeline/lsm_tree.rs:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::error::CoreError;
+use crate::storage::Storage;
+use crate::Result;
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+
+pub const LSM_TIMELINE_DIR: &str = ".hoodie/timeline";

Review Comment:
   Noted; i'll wire in the configs but archived loaders will actually resolve paths from configs in the next PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2452318890


##########
crates/core/src/timeline/loader.rs:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::internal::HudiInternalConfig::{
+    TimelineArchivedReadEnabled, TimelineArchivedUnavailableBehavior,
+};
+use crate::config::table::HudiTableConfig::ArchiveLogFolder;
+use crate::config::HudiConfigs;
+use crate::error::CoreError;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::lsm_tree::LSM_TIMELINE_DIR;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use log::debug;
+use std::sync::Arc;
+
+#[derive(Debug, Clone)]
+pub enum TimelineLoader {
+    LayoutOneActive(Arc<Storage>),
+    LayoutOneArchived(Arc<Storage>),
+    LayoutTwoActive(Arc<Storage>),
+    LayoutTwoArchived(Arc<Storage>),
+}
+
+impl TimelineLoader {
+    pub async fn load_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        match self {
+            TimelineLoader::LayoutOneActive(storage) => {
+                let files = storage.list_files(Some(HUDI_METADATA_DIR)).await?;
+                let mut instants = Vec::with_capacity(files.len() / 3);
+
+                for file_info in files {
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            TimelineLoader::LayoutTwoActive(storage) => {
+                let files = storage.list_files(Some(LSM_TIMELINE_DIR)).await?;
+                let mut instants = Vec::new();
+
+                for file_info in files {
+                    if file_info.name.starts_with("history/") || file_info.name.ends_with(".crc") {
+                        continue;
+                    }
+                    match selector.try_create_instant(file_info.name.as_str()) {
+                        Ok(instant) => instants.push(instant),
+                        Err(e) => {
+                            debug!(
+                                "Instant not created from file {:?} due to: {:?}",
+                                file_info, e
+                            );
+                        }
+                    }
+                }
+
+                instants.sort_unstable();
+                instants.shrink_to_fit();
+
+                if desc {
+                    Ok(instants.into_iter().rev().collect())
+                } else {
+                    Ok(instants)
+                }
+            }
+            _ => Err(CoreError::Unsupported(
+                "Loading from this timeline layout is not implemented yet.".to_string(),
+            )),
+        }
+    }
+
+    pub(crate) async fn load_archived_instants(
+        &self,
+        selector: &TimelineSelector,
+        desc: bool,
+    ) -> Result<Vec<Instant>> {
+        // Config wiring: if archived read not enabled, return behavior based on policy
+        let storage = match self {
+            TimelineLoader::LayoutOneArchived(storage)
+            | TimelineLoader::LayoutTwoArchived(storage) => storage.clone(),
+            _ => return Ok(Vec::new()), // Active loaders don't have archived parts
+        };
+
+        let configs: Arc<HudiConfigs> = storage.hudi_configs.clone();
+        let enabled = configs
+            .get_or_default(TimelineArchivedReadEnabled)
+            .to::<bool>();
+        if !enabled {
+            let behavior = configs
+                .get_or_default(TimelineArchivedUnavailableBehavior)
+                .to::<String>()
+                .to_ascii_lowercase();
+            return match behavior.as_str() {
+                "error" => Err(CoreError::Unsupported(
+                    "Archived timeline read is disabled; shorten time range or enable archived read"
+                        .to_string(),
+                )),
+                _ => Ok(Vec::new()), // continue silently with empty archived
+            };
+        }
+
+        match self {
+            TimelineLoader::LayoutOneArchived(storage) => {
+                // Resolve archive folder from configs or fallback
+                let archive_dir = configs
+                    .try_get(ArchiveLogFolder)
+                    .map(|v| v.to::<String>())
+                    .unwrap_or_else(|| ".hoodie/archived".to_string());

Review Comment:
   used `get_or_default`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#issuecomment-3066653102

   ## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/395?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `51.92308%` with `25 lines` in your changes missing coverage. Please review.
   > Project coverage is 88.35%. Comparing base [(`6bd998a`)](https://app.codecov.io/gh/apache/hudi-rs/commit/6bd998aca1c2cbcc6cdd3646530f8e2959207ccb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`98b19ae`)](https://app.codecov.io/gh/apache/hudi-rs/commit/98b19ae274d303106e608a340f68fcf56964cea2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/395?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [crates/core/src/timeline/lsm.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/395?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ftimeline%2Flsm.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3RpbWVsaW5lL2xzbS5ycw==) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/395?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [crates/core/src/timeline/layout.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/395?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ftimeline%2Flayout.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3RpbWVsaW5lL2xheW91dC5ycw==) | 66.66% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/395?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [crates/core/src/timeline/mod.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/395?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ftimeline%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3RpbWVsaW5lL21vZC5ycw==) | 55.55% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/395?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [crates/core/src/timeline/active.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/395?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ftimeline%2Factive.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3RpbWVsaW5lL2FjdGl2ZS5ycw==) | 85.71% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/395?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main     #395      +/-   ##
   ==========================================
   - Coverage   88.89%   88.35%   -0.54%     
   ==========================================
     Files          49       52       +3     
     Lines        2711     2749      +38     
   ==========================================
   + Hits         2410     2429      +19     
   - Misses        301      320      +19     
   ```
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/395?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2204765104


##########
crates/core/src/timeline/layout.rs:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::HudiConfigs;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use std::sync::Arc;
+
+/// Timeline layout implementations for different table versions.
+#[derive(Debug, Clone)]
+pub enum TimelineLayoutType {
+    Active(super::active::ActiveTimeline),
+    Lsm(super::lsm::LsmTimeline),

Review Comment:
   @codope cool! keeping lsm tree specific in `lsm_tree.rs` is good. may not need to worry about re-using it for now - it can be used by the loaders exclusively, but somewhat positioned for future re-use
   
   Looking closer - so `load_instants()` is actually a more generic helper function, it's the using of a `selector` in one of the init functions (`new_from_storage()`) that defines the behavior of timeline's instants loading. So depends on where `load_instants()` gets called, we just need to pass in the desired `TimelineSelector`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add LSM timeline support with timeline layout management for old and new table versions [hudi-rs]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #395:
URL: https://github.com/apache/hudi-rs/pull/395#discussion_r2204509388


##########
crates/core/src/timeline/layout.rs:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::config::HudiConfigs;
+use crate::storage::Storage;
+use crate::timeline::instant::Instant;
+use crate::timeline::selector::TimelineSelector;
+use crate::Result;
+use std::sync::Arc;
+
+/// Timeline layout implementations for different table versions.
+#[derive(Debug, Clone)]
+pub enum TimelineLayoutType {
+    Active(super::active::ActiveTimeline),
+    Lsm(super::lsm::LsmTimeline),

Review Comment:
   I agree, better suggestion overall! Few things if you can clarify:
   
   > We also avoid calling a LSM timeline since LSM tree is just one impl of Timeline which should be hidden away from Timeline's level
   
   Should the LSM tree logic be a separate module that the loaders use, or should the loaders themselves be the LSM tree implementations? So, in the context of `TimelineLoader::LayoutTwoActive`, would this variant contain an `LSMTree` instance, or would it implement LSM tree logic directly?
   
   Also, should the current `Timeline::load_instants()` method remain as-is, or would you prefer a different public API? For example, another API that takes a boolean to decide whether to load archived/compacted timeline. Alternatively, we could just add another API keeping `load_instants()` as-is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org